search

Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.98k stars 560 forks source link

parallel building with Win32 dmake problems #14518

Closed p5pRT closed 9 years ago

p5pRT commented 9 years ago

Migrated from rt.perl.org#123854 (status was 'resolved')

Searchable as RT123854$

p5pRT commented 9 years ago

From @bulk88

Created by @bulk88

This really shouldn't be a ticket\, but the mailing list server has a 250KB message size limit\, and I dont want to send 22 posts to the ML. The log files will be in a different post for this ticket since that post will bounce from the ML server when RT forwards it to the ML server.

I decided to implement parallel building with dmake since nmake's clone with parallelism\, QT's jom make tool\, has serious bugs\, and I dont think it is worth my time to get them fixed when dmake parallelism doesn't have straight up "build will absolutely fail" bugs. Jom's specific bug IIRC is doesn't support having a dep/target with ".." in it. A target with ".." in its name is silently skipped.

I am using dmake that came with strawberry perl 5.18. It calls itself dmake 4.12 ---------------------------------------------------------------- C​:\perl521>dmake -V dmake - Version 4.12 (Windows / MinGW) Copyright (c) 1990\,...\,1997 by WTI Corp.

Default Configuration​:   MAXLINELENGTH := 32766   MAXPROCESSLIMIT := 64   .IMPORT .IGNORE​: DMAKEROOT   .MAKEFILES : makefile.mk makefile   .SOURCE : .NULL   DMAKEROOT *= $(ABSMAKECMD​:d)startup   MAKESTARTUP := $(DMAKEROOT)\startup.mk

Please read the NEWS file for the latest release notes.

C​:\perl521> ----------------------------------------------------------------

But dmake parallel ("dmake -P2 all") has very bad design in finding "next shell line to run". A "dmake -P2 all" does around "!unsatisfied_deps_of_target_all" IO stat/file test calls to MS CRT/kernel\, for each recipe line. With procmon profiling\, in 1 minute\, it made 839447 IO calls (839447 lines\, 99% of which are IO calls). I've attached the IO call log in the 2nd post to this ticket. It contains about 1 minute (and remember\, each IO calls is slower because of the procmon profiler) of IO calls of dmake.exe. Yes it is 150 MB long. With file IO profiling off\, and just process events turned on in procmon\, it takes 3 SECONDS to run each "echo something >> config.h" line on the profilers. Without the procmon profiler\, it takes 30-35 seconds to generate config.h with 1 core maxed out at nearly 100% kernel time for that core. So without procmon\, it takes 350 ms to do each echo shell call. Below a console log of what is happening.

C​:\perl521\srcnewb4opt\win32>dmake -P2 all dmake​: makefile.mk​: line 1682​: Warning​: -- Macro `TESTPREPGCC' redefined afte r use if not exist ".\mini" mkdir ".\mini" if exist config.h del /f config.h copy config_H.vc config.h rem. > .\mini\.exists lib /def​:ntdllundec.def /machine​:x86 copy config.vc config.w32 Microsoft (R) Libra 1 file(s) copied. ry Manager Version 7.10.6030 Copyright (C) Microsoft Corporation. All rights reserved.

  Creating library ntdllundec.lib and object ntdllundec.exp lib /def​:ntdlldec.def /machine​:X86 ntdllundec.lib   1 file(s) copied. Microsoft (R) Library Manager Version 7.10.6030 Copyright (C) Microsoft Corporation. All rights reserved.

LIB : warning LNK4001​: no object files specified; libraries used   Creating library ntdlldec.lib and object ntdlldec.exp copy runperl.c perlmain.c rc -i.. -DINCLUDE_MANIFEST perlexe.rc   1 file(s) copied. cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DN O_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_TEXTMODE_S CRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T -Foperlgl ob.obj -Fdperlglob.pdb perlglob.c perlglob.c link -nologo -nodefaultlib -debug -opt​:ref\,icf -ltcg -libpath​:"c​:\per l\lib\CORE" -machine​:x86 oldnames.lib kernel32.lib user32.lib gdi32. lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib   version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib -out​:..\perl glob.exe -subsystem​:console \   perlglob.obj setargv.obj dmake​: Warning​: -- Target [..\perly.c] was made but the time stamp has not been   updated. if exist ..\perlglob.exe.manifest mt -nologo -manifest ..\perlglob.exe.manifest -outputresource​:..\perlglob.exe;1 && if exist ..\perlglob.exe.manif est del ..\perlglob.exe.manifest LINK : warning LNK4235​: /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance setargv.obj : warning LNK4099​: PDB 'libc.pdb' was not found with 'C​:\Program Fil es\Microsoft Visual Studio .NET 2003\VC7\LIB\setargv.obj' or at 'C​:\perl521\srcn ewb4opt\libc.pdb'; linking object as if no debug info ********everything above took 3 seconds*********************** *******30 to 35 second pause with no console output here\, this is generating miniperl's config.h with many echo >> config.h shell calls ******************* cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\av.obj -Fdmini\av.pdb ..\av.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\caretx.obj -Fdmini\caretx.pdb ..\caretx.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\deb.obj -Fdmini\deb.pdb ..\deb.c av.c caretx.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\doio.obj -Fdmini\doio.pdb ..\doio.c deb.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\doop.obj -Fdmini\doop.pdb ..\doop.c doio.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\dump.obj -Fdmini\dump.pdb ..\dump.c doop.c cl -c -I..\ -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOL E -DNO_STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_TE XTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T - Fo..\generate_uudmap.obj -Fd..\generate_uudmap.pdb ..\generate_uudmap.c dump.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\gv.obj -Fdmini\gv.pdb ..\gv.c generate_uudmap.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\mro.obj -Fdmini\mro.pdb ..\mro.c gv.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\hv.obj -Fdmini\hv.pdb ..\hv.c cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO _STRICT -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DNDEBUG -DPERL_EXTERNAL_G LOB -DPERL_IS_MINIPERL -Fomini\locale.obj -Fdmini\locale.pdb ..\locale.c mro.c hv.c ******killed******

By my eye\, "dmake -P2 ../mini/av.obj" is faster than "dmake -P2 ../miniperl.exe" is faster than "dmake -P2 Extensions" which is faster than "dmake -P2 all"\, all because of that delay before the first .obj for miniperl is produced. The narrower the build target\, or the less total targets\, the shorter that pause before the first miniperl .obj is compiled.

I am thinking that the only way to hack fix the performance problem with the echos is turn all the echo calls in .\config.h target http​://perl5.git.perl.org/perl.git/blob/HEAD​:/win32/makefile.mk#l1054 into 1 very long shell line filled with &&s. The other solution is for miniperl.exe and miniperl's config.h\, and buildcustomize.pl to be built from a 2nd dmake lang makefile\, to break the IO file existence test chain tree up. Anyone disagree to consolidating all the echos into 1 shell call or have any other opinions?

Perl Info ``` Flags: category=core severity=low Site configuration information for perl 5.21.4: Configured by Owner at Thu Sep 18 12:08:58 2014. Summary of my perl5 (revision 5 version 21 subversion 4) configuration: Derived from: 7d2b2edb94ab56333b9049a3e26d15ea18445512 Ancestor: 19be3be6968e2337bcdfe480693fff795ecd1304 Platform: osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -DWIN32 -D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T', optimize='-O1 -MD -Zi -DNDEBUG', cppflags='-DWIN32' ccversion='12.00.8168', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl521\lib\CORE" -machine:x86' libpth=C:\PROGRA~1\MIAF9D~1\VC98\lib libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl521\lib\CORE" -machine:x86' Locally applied patches: uncommitted-changes a0fe7a7e75de29e59f1da0d6822dc06e5be658fe a261faffee83d0145642ab5d1d046c9f813bc497 6506ab86ad1602a9ca720fcd30446dce1461d23d 7d2b2edb94ab56333b9049a3e26d15ea18445512 @INC for perl 5.21.4: lib C:/perl521/srcnew/lib . Environment for perl 5.21.4: HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH= PERL_BADLANG (unset) PERL_JSON_BACKEND=Cpanel::JSON::XS PERL_YAML_BACKEND=YAML SHELL (unset) ```
p5pRT commented 9 years ago

From @bulk88

[bad parallel dmake procs only.CSV](https://rt-archive.perl.org/perl5/Ticket/Attachment/1331679/711111/bad parallel dmake procs only.CSV)

p5pRT commented 9 years ago

From @bulk88

detailed IO log mentioned in OP

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 9 years ago

From @bulk88

[bad parallel dmake procs+disk IO.7z](https://rt-archive.perl.org/perl5/Ticket/Attachment/1331687/711117/bad parallel dmake procs+disk IO.7z)

p5pRT commented 9 years ago

From @steve-m-hay

[...]

I am thinking that the only way to hack fix the performance problem with the echos is turn all the echo calls in .\config.h target http​://perl5.git.perl.org/perl.git/blob/HEAD​:/win32/makefile.mk#l1054 into 1 very long shell line filled with &&s. The other solution is for miniperl.exe and miniperl's config.h\, and buildcustomize.pl to be built from a 2nd dmake lang makefile\, to break the IO file existence test chain tree up. Anyone disagree to consolidating all the echos into 1 shell call or have any other opinions?

I haven't looked through the details\, but I have no particular problem with your suggestion as long as the lines in the makefile.mk are still separate (i.e. using the \ continuation character) - one big long line in the makefile.mk would be gross!

Another option might be to put all the echo commands into a batch file in win32/ and have the makefile.mk issue a single shell call to run that batch file (passing in the required macro values as arguments)? That would have the additional benefit that the echo commands could be shared with win32/Makefile\, which has a very similar block of echoes in it. Or would that still cause trouble because the batch file is still doing separate echoes?

p5pRT commented 9 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 9 years ago

From @bulk88

On Tue Feb 17 00​:36​:33 2015\, shay wrote​:

I haven't looked through the details\, but I have no particular problem with your suggestion as long as the lines in the makefile.mk are still separate (i.e. using the \ continuation character) - one big long line in the makefile.mk would be gross!

This idea was done in the patch in https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 so this ticket can be closed if nobody else has objections.

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 9 years ago

From @tonycoz

On Tue Apr 28 12​:40​:02 2015\, bulk88 wrote​:

This idea was done in the patch in https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 so this ticket can be closed if nobody else has objections.

Done.

Tony

p5pRT commented 9 years ago

@tonycoz - Status changed from 'open' to 'resolved'