parallel building of core .c files on VC

[p5pRT]

Migrated from rt.perl.org#123482 (status was 'rejected')

Searchable as RT123482$

[p5pRT]

From @bulk88

Created by @bulk88

I tried playing around with parallel building (-MP[number]) option of cl.exe. -MP was added in VC 2008. A long time ago I tried using jmake or cmake (I dont remember) with VC 2003 and telling it to parallel build\, IIRC that failed since all the cl.exe'es tried to get file locks on "vc70.pdb" (a symbol file)\, and all but 1 cl.exe fatally exited. In that case\, 2 unapatizing solutions exist\, disabling .pdb creation (absolutly no for P5P\, unless its a smoke machine)\, or copying each .c file to its own folder and changing cwd before calling cl.exe. That way each cl.exe has their own vc70.pdb file simultanously. But these 2 solutions are for DIY parallism\, not MS sponsored parallelism. -MP on VC 2008 does produce multiple simultanous cl.exe processes but they dont seem to mind sharing 1 vc90.pdb file or 1 cwd\, so maybe MS fixed the file lock on a .pdb problem when they created -MP or -MP uses IPC or something to tell the child cl.exe'es to not lock the file exclusivly and share it with other child cl.exe'es but not foreign cl.exe processes (not lauched with -MP).

First attempt attempt failed and nothing was built. /Fo is global for some stupid reason instead of being bound to the .c right ahead of it on the command line.

C​:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D WIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE - D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:p recise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8 -Fomini\av.obj ..\av.c -Fom ini\caretx.obj ..\caretx.c cl : Command line warning D9025 : overriding '/Fomini\av.obj' with '/Fomini\care tx.obj' cl : Command line warning D9026 : options apply to entire command line cl : Command line error D8036 : '/Fomini\caretx.obj' not allowed with multiple s ource files

Attempt 2\, use -Fo in directory mode

C​:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D WIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE - D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:p recise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8 -Fomini ..\av.c ..\caretx.c

cl : Command line error D8036 : '/Fomini' not allowed with multiple source files

Attempt 3 I picked up on google (and an MS employee says they will try to fix "the bug" but put a backslash in the meantime) \, make -Fo end with a \

C​:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D WIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE - D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -GL -O1 -MD -Zi -DDEBUGGING - fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8  -Fomini\ ..\av.c  ..\c aretx.c  ..\deb.c  ..\doio.c  ..\doop.c  ..\dump.c  ..\globals.c  ..\gv.c  ..\mr o.c  ..\hv.c  ..\locale.c  ..\keywords.c  ..\mathoms.c  ..\mg.c  ..\numeric.c  . .\op.c  ..\pad.c  ..\perl.c  ..\perlapi.c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\ pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c  ..\reentr.c  ..\regcomp.c  .. \regexec.c  ..\run.c  ..\scope.c  ..\sv.c  ..\taint.c  ..\toke.c  ..\universal.c   ..\utf8.c  ..\util.c  ..\miniperlmain.c  ..\perlio.c

Successful build log from console attached as attachment.

The big problem now is\, how to implement this in make language? Since now many .c targets have to be built with 1 rule invocation so 1 cl.exe process runs. Also the dependency on     ..\generate_uudmap.exe  and ..\uudmap.h ..\bitcount.h ..\mg_data.h has to be tweaked so those 4 things are built before any core .c files are ever compiled. the nmake makefile currently has

$(MINIDIR)\globals$(o) : $(GENERATED_HEADERS)

so only global.c depends on those 3 files\, for parallel building an ifdef will be needed (remember we still support non-MP option VC) so all core .c files depend on those 3 headers.

Below is a "nmake -n" log

    del /f config.h     copy config_H.vc config.h     echo.>>.\config.h     echo #ifndef _config_h_footer_>>.\config.h     echo #define _config_h_footer_>>.\config.h     echo #undef Off_t>>.\config.h     echo #undef LSEEKSIZE>>.\config.h     echo #undef Off_t_size>>.\config.h     echo #undef PTRSIZE>>.\config.h     echo #undef SSize_t>>.\config.h     echo #undef HAS_ATOLL>>.\config.h     echo #undef HAS_STRTOLL>>.\config.h     echo #undef HAS_STRTOULL>>.\config.h     echo #undef IVTYPE>>.\config.h     echo #undef UVTYPE>>.\config.h     echo #undef IVSIZE>>.\config.h     echo #undef UVSIZE>>.\config.h     echo #undef NV_PRESERVES_UV>>.\config.h     echo #undef NV_PRESERVES_UV_BITS>>.\config.h     echo #undef IVdf>>.\config.h     echo #undef UVuf>>.\config.h     echo #undef UVof>>.\config.h     echo #undef UVxf>>.\config.h     echo #undef UVXf>>.\config.h     echo #undef USE_64_BIT_INT>>.\config.h     echo #undef Size_t_size>>.\config.h     echo #define Off_t __int64>>.\config.h     echo #define LSEEKSIZE ^8>>.\config.h     echo #define Off_t_size ^8>>.\config.h     echo #define PTRSIZE ^8>>.\config.h     echo #define SSize_t __int64>>.\config.h     echo #define HAS_ATOLL>>.\config.h     echo #define HAS_STRTOLL>>.\config.h     echo #define HAS_STRTOULL>>.\config.h     echo #define Size_t_size ^8>>.\config.h     echo #define IVTYPE __int64>>.\config.h     echo #define UVTYPE unsigned __int64>>.\config.h     echo #define IVSIZE ^8>>.\config.h     echo #define UVSIZE ^8>>.\config.h     echo #undef NV_PRESERVES_UV>>.\config.h     echo #define NV_PRESERVES_UV_BITS 53>>.\config.h     echo #define IVdf "I64d">>.\config.h     echo #define UVuf "I64u">>.\config.h     echo #define UVof "I64o">>.\config.h     echo #define UVxf "I64x">>.\config.h     echo #define UVXf "I64X">>.\config.h     echo #define USE_64_BIT_INT>>.\config.h     echo #endif>>.\config.h     if not exist ".\mini" mkdir ".\mini"     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\av.obj ..\av.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\caretx.obj ..\caretx.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\deb.obj ..\deb.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\doio.obj ..\doio.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\doop.obj ..\doop.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\dump.obj ..\dump.c     cl -c -I.. -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise  -DPERL_TEXTMODE_SCRIPTS -DPERL_HASH_FUNC_ONE_AT_A_TIME -DNO_MATHOMS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -Fo..\generate_uudmap.obj ..\generate_uudmap.c     link -subsystem​:console -out​:..\generate_uudmap.exe @​C​:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nmD65.tmp     if exist ..\generate_uudmap.exe.manifest mt -nologo -manifest ..\generate_uudmap.exe.manifest -outputresource​:..\generate_uudmap.exe;1 &&  if exist ..\generate_uudmap.exe.manifest del ..\generate_uudmap.exe.manifest     ..\generate_uudmap.exe ..\uudmap.h ..\bitcount.h ..\mg_data.h     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\globals.obj ..\globals.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\gv.obj ..\gv.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\mro.obj ..\mro.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\hv.obj ..\hv.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\locale.obj ..\locale.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\keywords.obj ..\keywords.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\mathoms.obj ..\mathoms.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\mg.obj ..\mg.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\numeric.obj ..\numeric.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\op.obj ..\op.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pad.obj ..\pad.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\perl.obj ..\perl.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\perlapi.obj ..\perlapi.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\perly.obj ..\perly.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp.obj ..\pp.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp_ctl.obj ..\pp_ctl.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp_hot.obj ..\pp_hot.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp_pack.obj ..\pp_pack.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp_sort.obj ..\pp_sort.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\pp_sys.obj ..\pp_sys.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\reentr.obj ..\reentr.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\regcomp.obj ..\regcomp.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\regexec.obj ..\regexec.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\run.obj ..\run.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\scope.obj ..\scope.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\sv.obj ..\sv.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\taint.obj ..\taint.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\toke.obj ..\toke.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\universal.obj ..\universal.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\utf8.obj ..\utf8.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\util.obj ..\util.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\miniperlmain.obj ..\miniperlmain.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\perlio.obj ..\perlio.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_IS_MINIPERL -Fo.\mini\win32.obj win32.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_IS_MINIPERL -Fo.\mini\win32sck.obj win32sck.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_IS_MINIPERL -Fo.\mini\win32thread.obj win32thread.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_IS_MINIPERL -Fo.\mini\fcrypt.obj fcrypt.c     cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_IS_MINIPERL -Fo.\mini\win32io.obj win32io.c     link -subsystem​:console -out​:..\miniperl.exe @​C​:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nmD66.tmp

Perl Info

``` Flags:     category=core     severity=low Site configuration information for perl 5.21.6: Configured by Administrator at Sat Oct 25 06:12:11 2014. Summary of my perl5 (revision 5 version 21 subversion 6) configuration:   Derived from:   Platform:     osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread     uname=''     config_args='undef'     hint=recommended, useposix=true, d_sigaction=undef     useithreads=define, usemultiplicity=define     use64bitint=undef, use64bitall=undef, uselongdouble=undef     usemymalloc=n, bincompat5005=undef   Compiler:     cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GS- -GL -DWIN32 -D_CONSOLE -DNO_STRICT -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE  -DPERL_TEXTMODE_SCRIPTS -DPERL_HASH_FUNC_ONE_AT_A_TIME -DNO_MATHOMS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO',     optimize='-O1 -MD -Zi -DNDEBUG -GS- -GL',     cppflags='-DWIN32'     ccversion='15.00.30729.01', gccversion='', gccosandvers=''     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8     alignbytes=8, prototype=define   Linker and Libraries:     ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"'     libpth=\lib     libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib     perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib     libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib     gnulibc_version=''   Dynamic Linking:     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '     cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"' Locally applied patches:     uncommitted-changes @INC for perl 5.21.6:     C:/p521/site/lib     C:/p521/lib     . Environment for perl 5.21.6:     CYGWIN=tty     HOME (unset)     LANG (unset)     LANGUAGE (unset)     LD_LIBRARY_PATH=/usr/lib/x86:/usr/X11R6/lib     LOGDIR (unset)     PATH=C:\p521\bin;C:\WINDOWS\system32;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC;C:\Program Files\TortoiseGit\bin;     PERL_BADLANG (unset)     SHELL (unset) ```

[p5pRT]

From @bulk88

Parallel build console output attached. Nothing very interesting in the file but that is what was in the console\, this is a win64 build so all the warnings are nothing unusual.

-- bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @bulk88

[parallel build console log.txt](https://rt-archive.perl.org/perl5/Ticket/Attachment/1323722/705661/parallel build console log.txt)

[p5pRT]

From @bulk88

More thoughts.

It seems the tool I tried a long time ago was not jmake or cmake but "jom" http​://qt-project.org/wiki/jom . The specific VC error is "fatal error C1033​: cannot open program database" http​://msdn.microsoft.com/en-us/library/4ks5fw35.aspx . This internet post https://social.msdn.microsoft.com/Forums/vstudio/en-US/a722c6b6-53ab-41a4-b6c1-5cb02b2623d4/visual-studio-2005-professional-mspdbsrv-failure-sporadic-fatal-error-c1033-cannot-open-program?forum=vcgeneral says to fix the conflict use -Fd http​://msdn.microsoft.com/en-us/library/9wst99a9%28v=vs.71%29.aspx . This is interesting since it means VC 6 and VC 2003 can be parallel without MS support\, and those 2 compilers are as fast as a mouse click in compiling C code (VC 2003 is the perfect blend of speed of compiling\, C lang support\, and good machine code emission\, for me)

Another issue is that the various .c file take different times to build. Building takes about 20 seconds on my 8 core Win64 machine with VC 2008 -O1 and DEBUGGING/asserts on but half (10 sec) of those 20 seconds are spent \<= 50% cpu on all 8 cores (1 thread can max out at 12.5%)\, so the parallelism isn't very great. It is 1​:30 to 2​:00 minutes without any parallelism (current way).

  C​:\perl\bin\perl.exe -E" @​a = qw( ..\av.c ..\caretx.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c ..\mathoms.c ..\mg.c ..\numeric.c ..\op.c ..\pad.c ..\perl.c ..\perlapi.c ..\perly.c ..\pp.c ..\pp_ctl.c ..\pp_hot.c ..\pp_pack.c ..\pp_sort.c ..\pp_sys.c ..\reentr.c ..\regcomp.c ..\regexec.c ..\run.c ..\scope.c ..\sv.c ..\taint.c ..\toke.c ..\universal.c ..\utf8.c ..\util.c ..\miniperlmain.c ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c); foreach(@​a) {my $t = time; system('cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fomini '.$_.' 1>NUL'); say $_.' '.(time-$t);}"

There are 3 very long to compile files.

They specifically are

..\regcomp.c 11 ..\regexec.c 9 ..\toke.c 15

raw data​:

..\av.c 1 ..\caretx.c 0 ..\deb.c 1 ..\doio.c 1 ..\doop.c 1 ..\dump.c 1 ..\globals.c 1 ..\gv.c 2 ..\mro.c 1 ..\hv.c 1 ..\locale.c 0 ..\keywords.c 2 ..\mathoms.c 1 ..\mg.c 1 ..\numeric.c 1 ..\op.c 3 ..\pad.c 1 ..\perl.c 1 ..\perlapi.c 1 ..\perly.c 1 ..\pp.c 2 ..\pp_ctl.c 3 ..\pp_hot.c 1 ..\pp_pack.c 4 ..\pp_sort.c 1 ..\pp_sys.c 1 ..\reentr.c 1 ..\regcomp.c 11 ..\regexec.c 9 ..\run.c 0 ..\scope.c 1 ..\sv.c 4 ..\taint.c 1 ..\toke.c 15 ..\universal.c 1 ..\utf8.c 1 ..\util.c 1 ..\miniperlmain.c 1 ..\perlio.c 1 win32.c 1 win32sck.c 1 win32thread.c 0 fcrypt.c 0 win32io.c 1

So that means unless someone will split ..\regcomp.c ..\regexec.c ..\toke.c into more .c files\, those 3 hold up every other .c file. I looked specifically in toke.c\, Perl_yylex is the largest function inside\, but there are a ton of S_ static functions\, so to split toke.c means creating static by use by not static by C language functions.

Top 5 largest by machine code from toke.c (big to small) Perl_yylex S_scan_const S_scan_str S_scan_heredoc S_scan_ident

Top 8 largest by machine code from regcomp.c (big to small) S_study_chunk S_reg S_make_trie S_regclass S_regatom S_join_exact S_dumpuntil Perl_regprop

Top 5 largest by machine code from regexec.c (big to small) S_find_byclass Perl_re_intuit_start S_regrepeat S_setup_EXACTISH_ST_c1_c2 S_reg_set_capture_string

With the pigs moved to the front of the cmd line

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\regcomp.c ..\regexec.c ..\toke.c ..\av.c ..\caret x.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c ..\mathoms.c ..\mg.c ..\numeric.c ..\op .c ..\pad.c ..\perl.c ..\perlapi.c ..\perly.c ..\pp.c ..\pp_ctl.c ..\pp_h ot.c ..\pp_pack.c ..\pp_sort.c ..\pp_sys.c ..\reentr.c ..\run.c ..\scope .c ..\sv.c ..\taint.c ..\universal.c ..\utf8.c ..\util.c ..\miniperlmain .c ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c 1>NUL && C​:\ perl\bin\perl.exe -E" say time" 1419301655 1419301673

18 seconds

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\regcomp.c ..\regexec.c ..\toke.c ..\av.c ..\caret x.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c ..\mathoms.c ..\mg.c ..\numeric.c ..\op .c ..\pad.c ..\perl.c ..\perlapi.c ..\perly.c ..\pp.c ..\pp_ctl.c ..\pp_h ot.c ..\pp_pack.c ..\pp_sort.c ..\pp_sys.c ..\reentr.c ..\run.c ..\scope .c ..\sv.c ..\taint.c ..\universal.c ..\utf8.c ..\util.c ..\miniperlmain .c ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c 1>NUL && C​:\ perl\bin\perl.exe -E" say time" 1419301928 1419301947

19 seconds

Now putting the fat ones on the end of the cmd line

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\av.c ..\caretx.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c   ..\mathoms.c ..\mg.c ..\numeric.c ..\op.c ..\pad.c ..\perl.c ..\perlapi. c ..\perly.c ..\pp.c ..\pp_ctl.c ..\pp_hot.c ..\pp_pack.c ..\pp_sort.c .. \pp_sys.c ..\reentr.c ..\run.c ..\scope.c ..\sv.c ..\taint.c ..\unive rsal.c ..\utf8.c ..\util.c ..\miniperlmain.c ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c ..\toke.c ..\regcomp.c ..\regexec.c 1>NUL && C :\perl\bin\perl.exe -E" say time" 1419301793 1419301816

23 seconds

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\av.c ..\caretx.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c   ..\mathoms.c ..\mg.c ..\numeric.c ..\op.c ..\pad.c ..\perl.c ..\perlapi. c ..\perly.c ..\pp.c ..\pp_ctl.c ..\pp_hot.c ..\pp_pack.c ..\pp_sort.c .. \pp_sys.c ..\reentr.c ..\run.c ..\scope.c ..\sv.c ..\taint.c ..\unive rsal.c ..\utf8.c ..\util.c ..\miniperlmain.c ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c ..\toke.c ..\regcomp.c ..\regexec.c 1>NUL && C :\perl\bin\perl.exe -E" say time" 1419301868 1419301892

24 seconds

Further optimizing the list of .c files by compile time yields no benefit (sv.c\, op.c and pp_ctl.c)

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\regcomp.c ..\regexec.c ..\toke.c ..\sv.c ..\pp_pac k.c ..\pp_ctl.c ..\op.c ..\av.c ..\caretx.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c   ..\mathoms.c ..\mg.c ..\numeric.c ..\op.c ..\pad.c ..\perl.c ..\perlapi. c ..\perly.c ..\pp.c ..\pp_hot.c ..\pp_pack.c ..\pp_sort.c ..\pp_sys.c ..\reentr.c ..\run.c ..\scope.c ..\taint.c ..\universal.c ..\utf8.c ..\util.c ..\miniperlmain.c ..\perlio.c win32.c win32sck.c win32thread.c fcryp t.c win32io.c 1>NUL && C​:\perl\bin\perl.exe -E" say time" 1419302166 1419302185

19 seconds

C​:\p521\srcnew\win32> C​:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 - DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL - DPERL_CORE -O1 -MD -Zi -DDEBUGGING -fp​:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_ MINIPERL -MP8 -Fomini\ ..\regcomp.c ..\regexec.c ..\toke.c ..\sv.c ..\pp_pac k.c ..\pp_ctl.c ..\op.c ..\av.c ..\caretx.c ..\deb.c ..\doio.c ..\doop.c ..\dump.c ..\globals.c ..\gv.c ..\mro.c ..\hv.c ..\locale.c ..\keywords.c   ..\mathoms.c ..\mg.c ..\numeric.c ..\op.c ..\pad.c ..\perl.c ..\perlapi. c ..\perly.c ..\pp.c ..\pp_hot.c ..\pp_pack.c ..\pp_sort.c ..\pp_sys.c ..\reentr.c ..\run.c ..\scope.c ..\taint.c ..\universal.c ..\utf8.c ..\util.c ..\miniperlmain.c ..\perlio.c win32.c win32sck.c win32thread.c fcryp t.c win32io.c 1>NUL && C​:\perl\bin\perl.exe -E" say time" 1419302206 1419302225

19 seconds

-- bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @cpansprout

On Mon Dec 22 18​:43​:45 2014\, bulk88 wrote​:

There are 3 very long to compile files.

They specifically are

..\regcomp.c 11 ..\regexec.c 9 ..\toke.c 15

I’m surprised that sv.c and op.c are not at the top of the list. When it comes to the size of the machine code\, they are up there.

$ ls -lS *.o -rw-r--r-- 1 sprout staff 1121108 Dec 22 14​:58 sv.o -rw-r--r-- 1 sprout staff 848728 Dec 22 14​:58 regcomp.o -rw-r--r-- 1 sprout staff 821100 Dec 22 14​:59 op.o -rw-r--r-- 1 sprout staff 820712 Dec 22 14​:58 opmini.o -rw-r--r-- 1 sprout staff 787508 Dec 22 14​:58 toke.o -rw-r--r-- 1 sprout staff 541252 Dec 22 14​:58 pp.o -rw-r--r-- 1 sprout staff 521436 Dec 22 14​:58 pp_ctl.o -rw-r--r-- 1 sprout staff 462488 Dec 22 14​:59 pp_sys.o -rw-r--r-- 1 sprout staff 333196 Dec 22 14​:59 regexec.o -rw-r--r-- 1 sprout staff 329644 Dec 22 16​:09 perl.o -rw-r--r-- 1 sprout staff 325612 Dec 22 14​:59 perlio.o -rw-r--r-- 1 sprout staff 321208 Dec 22 14​:58 perlmini.o -rw-r--r-- 1 sprout staff 316292 Dec 22 14​:58 pp_hot.o -rw-r--r-- 1 sprout staff 312820 Dec 22 14​:58 gv.o -rw-r--r-- 1 sprout staff 298040 Dec 22 14​:58 mg.o -rw-r--r-- 1 sprout staff 289180 Dec 22 14​:59 pp_pack.o -rw-r--r-- 1 sprout staff 282812 Dec 22 14​:58 util.o -rw-r--r-- 1 sprout staff 271304 Dec 22 14​:59 utf8.o -rw-r--r-- 1 sprout staff 243096 Dec 22 14​:58 dump.o -rw-r--r-- 1 sprout staff 212356 Dec 22 14​:58 hv.o -rw-r--r-- 1 sprout staff 206076 Dec 22 14​:59 doio.o -rw-r--r-- 1 sprout staff 192632 Dec 22 14​:59 mathoms.o -rw-r--r-- 1 sprout staff 172868 Dec 22 14​:59 universal.o -rw-r--r-- 1 sprout staff 172216 Dec 22 14​:58 pad.o -rw-r--r-- 1 sprout staff 162392 Dec 22 14​:58 scope.o -rw-r--r-- 1 sprout staff 115752 Dec 22 14​:59 pp_sort.o -rw-r--r-- 1 sprout staff 114900 Dec 22 14​:59 globals.o -rw-r--r-- 1 sprout staff 111688 Dec 22 14​:59 doop.o -rw-r--r-- 1 sprout staff 105836 Dec 22 14​:58 mro.o -rw-r--r-- 1 sprout staff 104712 Dec 22 14​:58 perly.o -rw-r--r-- 1 sprout staff 93836 Dec 22 14​:59 numeric.o -rw-r--r-- 1 sprout staff 85344 Dec 22 14​:58 av.o -rw-r--r-- 1 sprout staff 72772 Dec 22 14​:59 locale.o -rw-r--r-- 1 sprout staff 56168 Dec 22 16​:09 DynaLoader.o -rw-r--r-- 1 sprout staff 55428 Dec 22 14​:58 keywords.o -rw-r--r-- 1 sprout staff 43532 Dec 22 14​:59 taint.o -rw-r--r-- 1 sprout staff 42900 Dec 22 14​:59 deb.o -rw-r--r-- 1 sprout staff 38192 Dec 22 14​:58 reentr.o -rw-r--r-- 1 sprout staff 33620 Dec 22 19​:02 perlmain.o -rw-r--r-- 1 sprout staff 33044 Dec 22 14​:58 miniperlmain.o -rw-r--r-- 1 sprout staff 32228 Dec 22 14​:59 caretx.o -rw-r--r-- 1 sprout staff 31444 Dec 22 14​:58 run.o -rw-r--r-- 1 sprout staff 17624 Dec 22 08​:35 generate_uudmap.o -rw-r--r-- 1 sprout staff 184 Dec 22 14​:59 perlapi.o

--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @bulk88

On Mon Dec 22 19​:09​:27 2014\, sprout wrote​:

On Mon Dec 22 18​:43​:45 2014\, bulk88 wrote​:

There are 3 very long to compile files.

They specifically are

..\regcomp.c 11 ..\regexec.c 9 ..\toke.c 15

I’m surprised that sv.c and op.c are not at the top of the list. When it comes to the size of the machine code\, they are up there.

First note\, this is Win32 miniperl\, no perlio\, no threads/PERL_IMPLICIT_CONTEXT\, and no psuedofork/PERL_IMPLICIT_SYS. And the perl_clone (biggest func in sv.c) is CPP ifdef'ed out. Also I'm not sure if sv_dump is optimized away or not in dump.c\, since Win32 miniperl doesn't export any functions\, and therefore Devel​::Peek can't exist.

I think most of time is spent in the optimizer\, and the optimizer takes atleast 1 pass of O(n^2) or O(!n)\, AST nodes/bytecode ops to run. Alot of tiny functions are fast to optimize\, since in a couple plunges into the tree\, every node was covered. A very large function will cause the optimize to plunge repeatedly into the tree backwards and forwards from each node\, and most of the plunges/searches into the tree exhaust the maximum search window and nothing is optimized. Without a maximum search window or a walltime counter\, the CC will either run out of memory\, or take 2^64 years to compile/optimize the app. Intel C allows you to configure the maximum search windows\, so if you decide that perf is so important that it is fine to take a month of 100% CPU to compile the binary\, you can make that choice. In a video interview with a VC dev that MS posted\, the VC dev said Visual C when he first got on the team in the early 2000s "couldn't optimize between [C] statements"\, due to its legacy of running on 640 KB machines. In bulk88's experience\, VC will optimize between statements aslong as they are in 1 branch of linear unconditional code to execute. VC very rarely will optimize away a branch "if(val) lval = 5; else lval = 0;" it will turn into "lval = (--(val == 0)) & 5;". It won't convert C auto bools ("boolval = 1; if(boolval) {....}") into direct jump ops like GCC.

But VC's register allocator and common machine code de-duping is light years ahead of GCC. With GCC I often see\, 2 branches that goto/flow into 1 target\, they are the same C statements after optimizing\, they were the same statement more or less after CPP\, but use different volatile registers\, and therefore\, I guess were not de-duped into the target block. GCC also does this

eax = *(esp+0x50); eax += 5; *(esp+0x50) = eax; if(ecx) goto no_call; edx = *(esp+0x4C); *(esp+0x8) = edx; /* outgoing arg 2 */ eax = *(esp+0x50); *(esp+0x4) = eax; /* outgoing arg 1 */ *(esp+0x0) = esi; /* outgoing arg 0\, my_perl */ call Perl_something; no_call​:

VC would do

*(esp+0x50) += 5; if(ecx) goto no_call; push *(esp+0x4C); /* outgoing arg 2 */ push *(esp+0x50); /* outgoing arg 1 */ push esi; /* outgoing arg 0\, my_perl */ call Perl_something; esp += 0xC; no_call​:

Even if GCC doesn't want to use push op\, and its refusal to use x86 CISC operands as if its making ARM machine code\, it should have done

eax = *(esp+0x50); eax += 5; *(esp+0x50) = eax; if(ecx) goto no_call; edx = *(esp+0x4C); *(esp+0x8) = edx; /* outgoing arg 2 */ /* note the op on this line doesn't exist anymore */ *(esp+0x4) = eax; /* outgoing arg 1 */ *(esp+0x0) = esi; /* outgoing arg 0\, my_perl */ call Perl_something; no_call​:

-- bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @hvds

"bulk88 via RT" \perlbug\-followup@&#8203;perl\.org wrote​: :Top 8 largest by machine code from regcomp.c (big to small) :S_study_chunk

I've been working on this one\, to split up and refactor it as part of Yves' plan to improve regexp compilation. You can see the work in progress on branch hv/study_chunk.

I think there'll be some knock-on effects to simplify S_reg() further down the line\, but it's mostly unchanged so far.

Hugo

[p5pRT]

From @bulk88

On Mon Dec 22 15​:02​:03 2014\, bulk88 wrote​:

This is a bug report for perl from bulk88@​hotmail.com\, generated with the help of perlbug 1.40 running under perl 5.21.6.

----------------------------------------------------------------- [Please describe your issue here]

I tried playing around with parallel building (-MP[number]) option of cl.exe. -MP was added in VC 2008.

This ticket can be closed.

It is obsolete because of https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 and http​://perl5.git.perl.org/perl.git/shortlog/c2c7bda0885d19e01f0d998f22d2248d7663e957 . In the patches there were commited\, I did the opposite of what I proposed in this ticket. In this ticket I proposed to use Visual C's parallelism options\, but in the final patches I used maketool parallelism. Visual C parallelism has problems that it is only in VC 2008 and newer\, and does nothing for parallelism between the 3 heavy targets\, mktables and nonxs and XS module building all of which exceed creating libperl in time. VC -MP also doesn't help speed up Mingw GCC builds which are slower to compile than VC builds. I also think\, but am not sure\, that LTCG/WPO/GL is disabled when using -MP\, and that is an unacceptable tradeoff for VC builds (-Od/-O0 is the fastest way to compile perl core). The parallelism for core .c files on a optimize VC perl is actually poor\, since all the .c files compile in LTCG mode\, and each .c file takes \<= 1 second\, and then the real compiling happens during link time\, in 1 core\, in 1 thread. A non-parallel operation. For miniperl build time there is nothign else to do while compiling minipelr inside the VC linker\, but for full libperl\, there are plenty of other things to do in parallel like mktables and XS module building\, so I dont see the single threading of -GL being a problem.

-- bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @tonycoz

On Wed Sep 09 15​:53​:46 2015\, bulk88 wrote​:

On Mon Dec 22 15​:02​:03 2014\, bulk88 wrote​:

I tried playing around with parallel building (-MP[number]) option of cl.exe. -MP was added in VC 2008.

This ticket can be closed.

It is obsolete because of https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123867 and

Closed.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'rejected'