SHA.c: loadable library and perl binaries are mismatched

[shawnlaffan]

I get the error in the subject when there is a second perl in the path which has a different version. There is no error when there is only one perl in the path.

Tested on Windows using PAR::Packer 1.047 and PAR 1.015 (latest).

Steps to reproduce are below. These use portable Strawberry perls managed via berrybrew, but it also occurs when the second perl is a standard installation. The same happens with the order of perls reversed.

I have not been able to track down where it is coming from.

set PATH=C:\berrybrew\5.26.2_64_PDL\perl\site\bin;C:\berrybrew\5.26.2_64_PDL\perl\bin;C:\berrybrew\5.26.2_64_PDL\c\bin;C:\berrybrew\5.28.0_64_PDL\perl\site\bin;C:\berrybrew\5.28.0_64_PDL\perl\bin;C:\berrybrew\5.28.0_64_PDL\c\bin;%PATH%

pp -e "1"

An example result is:

SHA.c: loadable library and perl binaries are mismatched (got handshake key 0000000000028fa8, needed 0000000000000000)

If I run it multiple times then I get different handshake values, presumably due to different libraries being loaded.

Thanks, Shawn.

[rschupp]

The error is triggered from function Perl_xs_handshake in the Perl source util.c. I don't claim to understand the explanation given there and suspect that this a Windows specific problem.

Note that pp runs par.exe (extracted from PAR/StrippedPARL/Dynamic.pm or maybe .../Static.pm) and somewhere down the line Digest::SHA is loaded - that's where SHA.c referred to in the error message.

Also: which pp is run in your example?

[shawnlaffan]

Thanks Roderich,

pp and perl details for the error in the initial report are below.

As a further check to see if the build time search for dlls is the root cause, I built PAR::Packer with only the 5.28 perl in the path. However, the error persists when pp is called with a 5.26 perl later in the path.

Does par.exe search the path to load anything? If so then maybe it's going to the second instead of the first in the path? Otherwise I'll see if I can replicate using Digest::SHA.

Regards, Shawn.

pp -V
PAR Packager, version 1.047 (PAR version 1.015)

perl -v

This is perl 5, version 28, subversion 0 (v5.28.0) built for MSWin32-x64-multi-thread

[shawnlaffan]

Just an update, the error needs the second perl's bin directory to be in the path. The c/bin and site/bin directories have no effect.

Something is looking in the wrong perl/bin dir.

(This also corrects the path order from the original report - it should have been 5.28 and then 5.26, although the error occurs for either case).

set PATH=C:\berrybrew\5.28.0_64_PDL\c\bin;C:\berrybrew\5.28.0_64_PDL\perl\site\bin;C:\berrybrew\5.28.0_64_PDL\perl\bin;C:\berrybrew\5.26.11_64_PDL\perl\bin;%PATH%

[rschupp]

It may be relevant that you're using portable Strawberry perl installations. I think they are built with the relocatable option, i.e. any member of @INC that starts with ".../" has this prefix replaced with the dirname of the executable. And par (extracted to some temp file) inherits this behaviour.

[shawnlaffan]

It's looking likely that something in portable Strawberry perl is the root cause. The error does not occur when the first perl in the path is a standard installation but the second is portable. (Tested using PAR::Packer 1.045 and 1.047).

I'm not sure where it is picking up the wrong path name from, though. Dumping %Config::Config for a portable perl 5.28 first in the path shows nothing that refers to the 5.26 path. Neither does perl -V.

[rschupp]

If you copy a portable perl.exe to, e.g. %TMP%/foo.exe, what does running foo.exe -V report about @INC?

[shawnlaffan]

I copied two versions across, foo5.28.0.exe and foo5.26.2.exe.

The @INC entries are picking up the first matching install of strawberry perl. This is regardless of what else is in the path.

However, ldflags, lddflags and libpth all point to C:\strawberry. C:\strawberry is not in the path when I run these calls.

ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'
lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'
libpth=C:\strawberry\c\lib C:\strawberry\c\x86_64-w64-mingw32\lib C:\strawberry\c\lib\gcc\x86_64-w64-mingw32\7.1.0

The libperl entry is version specific: libperl=libperl528.a or libperl=libperl526.a.

Let me know if you want full perl -V dumps.

As a further point, C:\strawberry\perl is version 5.26.1, so one minor version earlier than the 5.26.2 I have been testing with. Copying it to %TMP%\foos5.26.1.exe gives the same results for perl -V, but this minor version difference might be related to the handshake error.

[shawnlaffan]

The problem goes away after renaming the C:\Strawberry directory to C:\Strawberryx. This is both for 5.26.2 and 5.28.0.

It also does not occur when the portable version is the same as the standard installation (5.26.1 in this case).

[rschupp]

Looking at ldflags, lddflags and libpth above I suspect that the problem is caused by a faulty build of PAR::Packer.

Can you rename C:\Strawberryx back to C:\Strawberry and attach the full log of building PAR::Packer with one of the portable installations.

[shawnlaffan]

where perl results in

C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe
C:\Strawberry\perl\bin\perl.exe

Perl -v:

This is perl 5, version 28, subversion 0 (v5.28.0) built for MSWin32-x64-multi-thread

I used cpanm --look PAR::Packer to download, then ran perl Makefile.PL and gmake. The result is below. I redirected stderr into stdout. Let me know if you want them separated.

cp lib/PAR/Filter/PatchContent.pm blib\lib\PAR\Filter\PatchContent.pm
cp lib/PAR/Filter/Bleach.pm blib\lib\PAR\Filter\Bleach.pm
cp lib/pp.pm blib\lib\pp.pm
cp lib/PAR/Filter/Bytecode.pm blib\lib\PAR\Filter\Bytecode.pm
cp lib/PAR/StrippedPARL/Base.pm blib\lib\PAR\StrippedPARL\Base.pm
cp lib/App/Packer/PAR.pm blib\lib\App\Packer\PAR.pm
cp lib/PAR/Packer.pm blib\lib\PAR\Packer.pm
cp lib/PAR/Filter/Obfuscate.pm blib\lib\PAR\Filter\Obfuscate.pm
cp lib/PAR/Filter/PodStrip.pm blib\lib\PAR\Filter\PodStrip.pm
cp lib/PAR/Filter.pm blib\lib\PAR\Filter.pm
gmake[1]: Entering directory 'C:/BERRYB~1/528~1.0_6/data/.cpanm/work/1539677732.3612/PAR-Packer-1.047/myldr'
Makefile:868: warning: overriding recipe for target '.c.o'
Makefile:335: warning: ignoring old recipe for target '.c.o'
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" par_pl2c.pl my_par_pl < ..\script\par.pl > my_par_pl.c 
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" sha1.c.PL
gcc -c -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields  -I"C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE"  -DLDLIBPTHNAME=\"\" -DPARL_EXE=\"parl.exe\" -DPAR_PACKER_VERSION=\"1.047\" -s -O2 main.c
windres -i winres/pp.rc -o ppresource.coff --input-format=rc --output-format=coff --target=pe-x86-64
g++.exe main.o ppresource.coff -s   -s -L"C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE" -L"C:\berrybrew\5.28.0_64_PDL\c\lib"  "C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE\libperl528.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libmoldname.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libkernel32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libuser32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libgdi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libwinspool.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libcomdlg32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libadvapi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libshell32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libole32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\liboleaut32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libnetapi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libuuid.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libws2_32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libmpr.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libwinmm.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libversion.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libodbc32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libodbccp32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libcomctl32.a"  -o par.exe
rem
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" encode_append.pl Dynamic.in par.exe Dynamic.pm
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" embed_files.pl -c 32768 par.exe recursive_objdump "C:\berrybrew\5.28.0_64_PDL\perl\bin\perl528.dll" > boot_embedded_files.c
# using method recursive_objdump to find files to embed
# embedding "par.exe" as "par.exe"
# embedding "C:\berrybrew\5.28.0_64_PDL\perl\bin/libstdc++-6.dll" as "libstdc++-6.dll"
# embedding "C:\berrybrew\5.28.0_64_PDL\perl\bin/libwinpthread-1.dll" as "libwinpthread-1.dll"
# embedding "C:\berrybrew\5.28.0_64_PDL\perl\bin/libgcc_s_seh-1.dll" as "libgcc_s_seh-1.dll"
# embedding "C:\berrybrew\5.28.0_64_PDL\perl\bin/perl528.dll" as "perl528.dll"
gcc -c -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields  -I"C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE"  -DLDLIBPTHNAME=\"\" -DPARL_EXE=\"parl.exe\" -DPAR_PACKER_VERSION=\"1.047\" -s -O2 boot.c
In file included from mktmpdir.h:87:0,
                 from mktmpdir.c:1,
                 from boot.c:10:
sha1.c: In function 'sha_transform':
sha1.c:146:4: warning: right shift count >= width of type [-Wshift-count-overflow]
  T >>= 32;
    ^~~
g++.exe boot.o  -static-libgcc -s   -s -L"C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE" -L"C:\berrybrew\5.28.0_64_PDL\c\lib"  "C:\berrybrew\5.28.0_64_PDL\perl\lib\CORE\libperl528.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libmoldname.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libkernel32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libuser32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libgdi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libwinspool.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libcomdlg32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libadvapi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libshell32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libole32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\liboleaut32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libnetapi32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libuuid.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libws2_32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libmpr.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libwinmm.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libversion.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libodbc32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libodbccp32.a" "C:\berrybrew\5.28.0_64_PDL\c\x86_64-w64-mingw32\lib\libcomctl32.a"  ppresource.coff -o boot.exe
rem
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" encode_append.pl Static.in boot.exe Static.pm
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -Mblib=.. run_with_inc.pl boot.exe -q -B -Oparl.exe
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -MExtUtils::Command -e cp -- parl.exe ..\blib\script\parl.exe
rem ..\blib\script\parl.exe
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -Mblib=.. run_with_inc.pl par.exe -q -B -Oparldyn.exe
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -MExtUtils::Command -e cp -- parldyn.exe ..\blib\script\parldyn.exe
rem ..\blib\script\parldyn.exe
gmake[1]: Leaving directory 'C:/BERRYB~1/528~1.0_6/data/.cpanm/work/1539677732.3612/PAR-Packer-1.047/myldr'
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -MExtUtils::Command -e cp -- script/par.pl blib\script\par.pl
pl2bat.bat blib\script\par.pl
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -MExtUtils::Command -e cp -- script/pp blib\script\pp
pl2bat.bat blib\script\pp
"C:\berrybrew\5.28.0_64_PDL\perl\bin\perl.exe" -MExtUtils::Command -e cp -- script/tkpp blib\script\tkpp
pl2bat.bat blib\script\tkpp

[shawnlaffan]

There is no meaningful difference in the build log when C:\Strawberry is renamed to c:\Strawberryx.

All that changes is the order of file copies and embedding operations.

[rschupp]

Yeah, the build log looks OK - no contamination with the stuff in C:\Strawberry

[shawnlaffan]

Given the foo.exe results, is there something in the packing sequence where a foo.exe equivalent loses track of the perl it is from and points to C:\strawberry?

[rschupp]

Yes, though I dunno how. The require Digest::SHA on line 605 of script/par.pl executed by par.exe (build from the portable installation) sometimes is called with

@INC = qw( C:\strawberry\perl\lib C:\strawberry\perl\lib C:\strawberry\perl\site\lib C:\strawberry\perl\site\lib C:\strawberry\perl\vendor\lib C:\strawberry\perl\vendor\lib )

Adding Digest::SHA to the list in require_modules() makes this module available in the cache area and hence works around this problem.

Shawn, can you try with HEAD from Git (you'll have to rebuild PAR::Packer for your portable installations)?

[shawnlaffan]

Thanks Roderich, that works on my system.