search

Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.9k stars 540 forks source link

[PATCH] re-parallelize Win32 build after Unicode::Normalize got XS #15029

Closed p5pRT closed 8 years ago

p5pRT commented 8 years ago

Migrated from rt.perl.org#126564 (status was 'resolved')

Searchable as RT126564$

p5pRT commented 8 years ago

From @bulk88

Created by @bulk88

time savings are 32-35 seconds

cmd​: timeit dmake -P8 CCTYPE=MSVC90 WIN64=undef

on 8 core machine

before at SHA-1​: a4a12576beffb0a87af8e541b0cbe0b9e1d10c9f

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        4​:34 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:07​:17.234 Process Time​:     0​:00​:13.000 System Calls​:     11424314 Context Switches​: 3657143 Page Faults​:      7185286 Bytes Read​:       1942259067 Bytes Written​:    263528445 Bytes Other​:      44141202

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        4​:41 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:07​:18.687 Process Time​:     0​:00​:13.234 System Calls​:     14659683 Context Switches​: 3735933 Page Faults​:      7417994 Bytes Read​:       2083261456 Bytes Written​:    274196571 Bytes Other​:      45311782

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        8​:14 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:07​:17.375 Process Time​:     0​:00​:13.218 System Calls​:     11037964 Context Switches​: 3629259 Page Faults​:      7120016 Bytes Read​:       1942149191 Bytes Written​:    263508177 Bytes Other​:      41817370

after this patch

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        7​:26 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:06​:45.953 Process Time​:     0​:00​:13.062 System Calls​:     10933492 Context Switches​: 3526741 Page Faults​:      7101234 Bytes Read​:       1941118763 Bytes Written​:    267130765 Bytes Other​:      40910193

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        7​:38 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:06​:44.859 Process Time​:     0​:00​:13.187 System Calls​:     11013682 Context Switches​: 3539636 Page Faults​:      7132940 Bytes Read​:       1942604418 Bytes Written​:    262682634 Bytes Other​:      41031853

Version Number​:   Windows NT 5.2 (Build 3790) Exit Time​:        8​:00 pm\, Wednesday\, November 4 2015 Elapsed Time​:     0​:06​:44.203 Process Time​:     0​:00​:12.968 System Calls​:     10910056 Context Switches​: 3504108 Page Faults​:      7089701 Bytes Read​:       1941547697 Bytes Written​:    263389475 Bytes Other​:      40789584

Perl Info ``` Flags:     category=core     severity=low Site configuration information for perl 5.21.8: Configured by Administrator at Thu Dec 25 21:47:06 2014. Summary of my perl5 (revision 5 version 21 subversion 8) configuration:   Derived from: a2809ed3f018e1bc7a25293a24e20f1045f13f63   Ancestor: 5e754124df5a29d96dd9bb3ecb0e73b5c11cd93a   Platform:     osname=MSWin32, osvers=5.2, archname=MSWin32-x64-multi-thread     uname=''     config_args='undef'     hint=recommended, useposix=true, d_sigaction=undef     useithreads=define, usemultiplicity=define     use64bitint=define, use64bitall=undef, uselongdouble=undef     usemymalloc=n, bincompat5005=undef   Compiler:     cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GS- -fp:precise -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE  -DPERL_TEXTMODE_SCRIPTS -DPERL_HASH_FUNC_ONE_AT_A_TIME -DNO_MATHOMS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO',     optimize='-O1 -MD -Zi -DNDEBUG -GS- -fp:precise',     cppflags='-DWIN32'     ccversion='15.00.30729.01', gccversion='', gccosandvers=''     intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0     ivtype='__int64', ivsize=8, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8     alignbytes=8, prototype=define   Linker and Libraries:     ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:AMD64 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"'     libpth=\lib     libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib     perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib     libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib     gnulibc_version=''   Dynamic Linking:     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '     cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:AMD64 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"' Locally applied patches:     uncommitted-changes     cd6c09c1d3b5d28e942d90a3395318bcf61f8efc     78c46fd42b26e41d1b756875baed4b0bd3581817     bf54a2e14d917af00e68d164996d31222b2ba840     dd9d34e232c848de58b389b6f312243628fe5d4e     b067b36797cfabaf9dc0520e7bef59d127ae2376     4a9d29b96c9c423676975c58c8df54f4d754d77b     8b25fc460db0bf51de2e56e68a5e1579cc76844a     3e71eedac4745e96e8467a9835d71d553b4a3362     afecf120eec0ee3460d131d898fb6f5ac20e42fc     0b97f3dc66b4d1dea20d8e27e764943f8960465b     ad27b3745cc703d543d4d4546a1af3a060b02e06     a2809ed3f018e1bc7a25293a24e20f1045f13f63 @INC for perl 5.21.8:     C:/p523/site/lib     C:/p523/lib     . Environment for perl 5.21.8:     CYGWIN=tty     HOME (unset)     LANG (unset)     LANGUAGE (unset)     LD_LIBRARY_PATH=/usr/lib/x86:/usr/X11R6/lib     LOGDIR (unset)     PATH=C:\p523\bin;C:\WINDOWS\system32;c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64;c:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE;C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\x64;C:\WINDOWS;C:\Program Files\TortoiseGit\bin;C:\sp3220\c\bin;     PERL_BADLANG (unset)     SHELL (unset) ```
p5pRT commented 8 years ago

From @bulk88

0001-re-parallelize-Win32-build-after-Unicode-Normalize-g.patch ```diff From 43fc48848d826dd093ab991c5185526155cdf499 Mon Sep 17 00:00:00 2001 From: Daniel Dragan Date: Wed, 4 Nov 2015 22:34:54 -0500 Subject: [PATCH] re-parallelize Win32 build after Unicode::Normalize got XS commit 57dca39807 serialized the 1st and 3rd longest targets (Extensions and UNIDATAFILES aka mktables) together after commit c6b7cc2176 which upgraded Unicode::Normalize, and Unicode::Normalize went from PP->XS, and the XS version requires unicore/CombiningClass.pl in cpan/Unicode-Normalize/mkheader during U::N's building. The serialization added about 32 second to a "dmake -P8 all" on a 8 core machine. Fix this by spltting off U::N from Extensions and put it in its own target. Making Extensions, the longest target slightly shorter by removing 1 module from its workload is another plus. See RT ticket for time savings details. --- win32/makefile.mk | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/win32/makefile.mk b/win32/makefile.mk index 5bdba0a..a6f46a3 100644 --- a/win32/makefile.mk +++ b/win32/makefile.mk @@ -1064,7 +1064,7 @@ CFG_VARS = \ # Top targets # -all : CHECKDMAKE rebasePE $(UNIDATAFILES) Extensions_nonxs $(PERLSTATIC) +all : CHECKDMAKE rebasePE Extensions_nonxs $(PERLSTATIC) ..\regcomp$(o) : ..\regnodes.h ..\regcharclass.h @@ -1457,8 +1457,11 @@ $(PERLEXESTATIC): $(PERLSTATICLIB) $(CONFIGPM) $(PERLEXEST_OBJ) $(PERLEXE_RES) # DynaLoader.pm, so this will have to do #most of deps of this target are in DYNALOADER and therefore omitted here -Extensions : $(PERLDEP) $(DYNALOADER) $(GLOBEXE) $(UNIDATAFILES) - $(MINIPERL) -I..\lib ..\make_ext.pl "MAKE=$(PLMAKE)" --dir=$(CPANDIR) --dir=$(DISTDIR) --dir=$(EXTDIR) --dynamic +Extensions : $(PERLDEP) $(DYNALOADER) $(GLOBEXE) + $(MINIPERL) -I..\lib ..\make_ext.pl "MAKE=$(PLMAKE)" --dir=$(CPANDIR) --dir=$(DISTDIR) --dir=$(EXTDIR) --dynamic !Unicode/Normalize + +Extensions_normalize : $(PERLDEP) $(DYNALOADER) $(GLOBEXE) $(UNIDATAFILES) + $(MINIPERL) -I..\lib ..\make_ext.pl "MAKE=$(PLMAKE)" --dir=$(CPANDIR) --dir=$(DISTDIR) --dir=$(EXTDIR) --dynamic +Unicode/Normalize Extensions_reonly : $(PERLDEP) $(DYNALOADER) $(MINIPERL) -I..\lib ..\make_ext.pl "MAKE=$(PLMAKE)" --dir=$(CPANDIR) --dir=$(DISTDIR) --dir=$(EXTDIR) --dynamic +re @@ -1484,9 +1487,9 @@ Extensions_realclean : # be running in parallel like UNIDATAFILES, this target a placeholder for the # future .IF "$(BUILD_STATIC)"=="define" -rebasePE : Extensions $(PERLDLL) $(PERLEXE) $(GLOBEXE) $(PERLEXESTATIC) +rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(GLOBEXE) $(PERLEXESTATIC) .ELSE -rebasePE : Extensions $(PERLDLL) $(PERLEXE) $(GLOBEXE) +rebasePE : Extensions $(PERLDLL) Extensions_normalize $(PERLEXE) $(GLOBEXE) .ENDIF $(NOOP) -- 1.8.0.msysgit.0 ```
p5pRT commented 8 years ago

From @steve-m-hay

Thanks\, applied as e598d4db5a36150a31a4fd4c97f8feff6cc6bf65.

p5pRT commented 8 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 8 years ago

@steve-m-hay - Status changed from 'open' to 'resolved'