If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. - Githubissues

perl11 / cperl

A perl5 with classes, types, compilable, company friendly, security

http://perl11.org/

Other

142 stars 17 forks source link

If ExifTool is used, binary compiled using perlcc depends on Config.so and cwd.so, even if --static/--staticxs is used. #423

Open jiucenglou opened 4 years ago

jiucenglou commented 4 years ago

If ExifTool is used, binary compiled using perlcc depends on Config.so and Cwd.so, even if --static/--staticxs is used. The reason might be that ExifTool utilizes Config.so . The problem is that the binary compiled requires cperl installation and is not portable any more. That is to say, the binary cannot run on another machine without cperl at the expected position.

Output of strace is shown below

stat("/usr/local/lib/cperl/5.30.0/x86_64-linux/auto/Config/Config.so", 0x7f4f8d872ba0) = -1 ENOENT (No such file or directory)
write(2, "Error: load_file /usr/local/lib/"..., 90Error: load_file /usr/local/lib/cperl/5.30.0/x86_64-linux/auto/Config/Config.so not found
) = 90

Could you suggest how to build the "true" static binary under this circumstance ? Many thanks !

rurban commented 4 years ago

A true static binary needs the Perl to be compiled static, with all modules be compiled static also. It is possible, but I never heard that anyone did that.

jiucenglou commented 4 years ago

Is it possible to compile everything statically base on the install instruction for cperl (shown below) ? :D If yes, could you suggest which switches one should use ? Many thanks !

http://perl11.org/cperl/

INSTALL:

Simple:

    ./Configure -sder -Dusedevel -Dusecperl
    make -s -j4
    make -s -j4 test
    sudo make install

Experts:

    ./Configure -sder -Dusedevel -Dusecperl \
      -Accflags='-msse4.2 -DPERL_FAKE_SIGNATURE' --optimize='-O3 -g' \
      -Dinstallman1dir=none -Dinstallman3dir=none -Dinstallsiteman1dir=none \
      -Dinstallsiteman3dir=none
    make -s -j4 ECHO=true
    make -s -j4 ECHO=true test
    sudo make install

vadimkantorov commented 4 years ago

Also interested if one can compile cperl statically with its modules - for the goal of embedding the interpreter statically to run some scripts, similar to what might (?) be accomplished with staticperl. Maybe it even can be a lightweight alternative to WebPerl and compile fully into WebAssembly. It would also be very useful for perlcc

vadimkantorov commented 4 years ago

Here's how I tried configuring cperl to compile statically: bash +x ./Configure -sde -Uusedl -Dprefix=$PREFIX log.zip

On my system it breaks with:

Run make depend now? [y]
`sh  cflags "optimize='-O3 --pipe'" generate_uudmap.o`  generate_uudmap.c
ld -o generate_uudmap -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe generate_uudmap.o -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
ld: unrecognized option '--pipe'
ld: use the --help option for usage information
Makefile:367: recipe for target 'generate_uudmap' failed
make: *** [generate_uudmap] Error 1

vadimkantorov commented 4 years ago

@rurban I tried to compile cperl with all modules in static way: -Dstatic_ext="B B/C Compress/Raw/Bzip2 Compress/Raw/Zlib Config Cpanel/JSON/XS Cwd Data/Dumper Devel/NYTProf Devel/PPPort Devel/Peek Digest/MD5 Digest/SHA Encode Fcntl File/DosGlob File/Glob Filter/Util/Call Hash/Util Hash/Util/FieldHash I18N/Langinfo IO IPC/SysV Internals/DumpArenas List/Util MIME/Base64 Math/BigInt/FastCalc Opcode POSIX PerlIO/encoding PerlIO/mmap PerlIO/scalar PerlIO/via SDBM_File Socket Storable Sys/Hostname Sys/Syslog Term/ReadKey Tie/Hash/NamedCapture Time/HiRes Time/Piece Unicode/Collate Unicode/Normalize XS/APItest XS/Typemap YAML/Safe mro re threads threads/shared"

At the end of compilation (linking time of the final cperl executable?) I got this error:

cc -o perl -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe -Wl,-E perlmain.o  lib/auto/B/B.a lib/auto/B/C/C.a lib/auto/Compress/Raw/Bzi
p2/Bzip2.a lib/auto/Compress/Raw/Zlib/Zlib.a lib/auto/Config/Config.a lib/auto/Cpanel/JSON/XS/XS.a lib/auto/Cwd/Cwd.a lib/auto/Data/Dumper/Dumper.a 
lib/auto/Devel/NYTProf/NYTProf.a lib/auto/Devel/PPPort/PPPort.a lib/auto/Devel/Peek/Peek.a lib/auto/Digest/MD5/MD5.a lib/auto/Digest/SHA/SHA.a lib/auto/Encode/Encode.a lib/auto/Fcntl/Fcntl.a lib/auto/File/DosGlob/DosGlob.a lib/auto/File/Glob/Glob.a lib/auto/Filter/Util/Call/Call.a lib/auto/Hash/Util/Util.a lib/auto/Hash/Util/FieldHash/FieldHash.a lib/auto/I18N/Langinfo/Langinfo.a lib/auto/IO/IO.a lib/auto/IPC/SysV/SysV.a lib/auto/Internals/DumpArenas/DumpArenas.a lib/auto/List/Util/Util.a lib/auto/MIME/Base64/Base64.a lib/auto/Math/BigInt/FastCalc/FastCalc.a lib/auto/Opcode/Opcode.a lib/auto/POSIX/POSIX.a lib/auto/PerlIO/encoding/encoding.a lib/auto/PerlIO/mmap/mmap.a lib/auto/PerlIO/scalar/scalar.a lib/auto/PerlIO/via/via.a lib/auto/SDBM_File/SDBM_File.a lib/auto/Socket/Socket.a lib/auto/Storable/Storable.a lib/auto/Sys/Hostname/Hostname.a lib/auto/Sys/Syslog/Syslog.a lib/auto/Term/ReadKey/ReadKey.a lib/auto/Tie/Hash/NamedCapture/NamedCapture.a lib/auto/Time/HiRes/HiRes.a lib/auto/Time/Piece/Piece.a lib/auto/Unicode/Collate/Collate.a lib/auto/Unicode/Normalize/Normalize.a lib/auto/XS/APItest/APItest.a lib/auto/XS/Typemap/Typemap.a lib/auto/YAML/Safe/Safe.a lib/auto/mro/mro.a lib/auto/re/re.a lib/auto/threads/threads.a lib/auto/threads/shared/shared.a lib/auto/Encode/Byte/Byte.a lib/auto/Encode/CN/CN.a lib/auto/Encode/EBCDIC/EBCDIC.a lib/auto/Encode/JP/JP.a lib/auto/Encode/KR/KR.a lib/auto/Encode/Symbol/Symbol.a lib/auto/Encode/TW/TW.a lib/auto/Encode/Unicode/Unicode.a libperl.a `cat ext.libs` -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isExclusion'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isSingleton'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isNonStDecomp'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
utf8.o (symbol from plugin): In function `nonchar_cp_format':
(.text+0x0): multiple definition of `isComp2nd'
Normalize.o (symbol from plugin):(.text+0x0): first defined here
../../perl.h:7634:19: warning: type of ‘PL_nan’ does not match original declaration [-Wlto-type-mismatch]
perl.h:7602:19: note: ‘PL_nan’ was previously declared here
 INFNAN_U8_NV_DECL PL_nan = { { DOUBLENANBYTES } };

This is not so many errors, so hopefully can be fixed easily

vadimkantorov commented 4 years ago

Maybe it's Unicode/Normalize module to blame. I'll try to confirm.

PL_nan warning is concerning though: it seems that two ifdef branches trigger on my system (WSLv1, Ubuntu 18.04) and it gets defined twice:

rurban commented 4 years ago

The duplicate Normalize symbols are easy to fix, and should be in a seperate ticket.

The nan stuff is an upstream problem, and indeed a severe bug

vadimkantorov commented 4 years ago

Also, regular expression module still got put into some dynamic_ext_re

vadimkantorov commented 4 years ago

A more lightweight setup:

./Configure -sde -Dprefix=$PREFIX -Dstatic_ext="Encode Config IO Fcntl Cwd Storable Digest/MD5 List/Util Time/HiRes File/Glob PerlIO/scalar"
bash +x Makefile.SH
make -j4
# succeeds

# remove installman line
make install
# fails

make[1]: Leaving directory '/mnt/c/Users/user/wiptlmgr/cperl-5.30.0/cpan/Devel-NYTProf'
Manifying 1 pod document
make[1]: Leaving directory '/mnt/c/Users/user/wiptlmgr/cperl-5.30.0/cpan/Math-BigInt-FastCalc'
./perl -Ilib -I. -f pod/buildtoc -q

        Everything is up to date. Type 'make test' to run test suite.
./perl -Ilib -I. installperl --destdir=
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Invalid version format (non-numeric data) at lib/File/Glob.pm line 68.
Compilation failed in require at installperl line 394.
BEGIN failed--compilation aborted at installperl line 394.
Makefile:589: recipe for target 'install-all' failed
make: *** [install-all] Error 255

lib/File/Glob.pm:68 has XSLoader::load();

vadimkantorov commented 4 years ago

What is XS and how do I link it statically?

vadimkantorov commented 4 years ago

Basically, static perl compiles fine (except for NaN issue), but Glob module fails:

vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/wiptlmgr/cperl-5.30.0$ ./perl -Ilib -I. installperl --destdir=
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Use of uninitialized value in XS subroutine entry at lib/File/Glob.pm line 68.
Invalid version format (non-numeric data) at lib/File/Glob.pm line 68.
Compilation failed in require at installperl line 394.
BEGIN failed--compilation aborted at installperl line 394.

vadimkantorov commented 4 years ago

It seems that the reason is that XSLoader module and XSLoader::load() doesn't work well.

I found XSLoader in Makefile:

DL_SRC = ext/DynaLoader/DynaLoader.xs ext/DynaLoader/dlboot.c ext/DynaLoader/dlutils.c          ext/DynaLoader/XSLoader.c

So it seems it still wants to do dynamic loading for something. What does it try to load dynamically for globbinb? libc?

vadimkantorov commented 4 years ago

Here lives that Glob.pm file: https://github.com/perl11/cperl/tree/0f5b639e779cc36fc203afefe79c7810906fac65/ext/File-Glob

Is XSLoader FFI supposed to support XSLoader::load() even if https://github.com/perl11/cperl/blob/0f5b639e779cc36fc203afefe79c7810906fac65/ext/File-Glob/Glob.xs is compiled statically?

Disalbing usedl completely, unfortunately leads to:

/bin/ln -s xsutils.c xsutilsmini.c
`sh  cflags "optimize='-O3 --pipe'" regexec.o`  regexec.c
`sh  cflags "optimize='-O3 --pipe'" utf8.o`  utf8.c
`sh  cflags "optimize='-O3 --pipe'" universal.o`  universal.c
`sh  cflags "optimize='-O3 --pipe'" perlapi.o`  perlapi.c
ld -o generate_uudmap -fstack-protector-strong -L/usr/local/lib -flto=4 -O3 --pipe generate_uudmap.o -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
ld: unrecognized option '--pipe'
ld: use the --help option for usage information
Makefile:367: recipe for target 'generate_uudmap' failed

vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/wiptlmgr$ /usr/bin/ld --version
GNU ld (GNU Binutils for Ubuntu) 2.30
Copyright (C) 2018 Free Software Foundation, Inc.

vadimkantorov commented 4 years ago

It seems that all static modules that use XSLoader have this problem: during make install or just in runtime. Here's a log of a similar problem of PerlIO/Scalar while running perlcc:

/mnt/c/Users/user/wiptlmgr/cperlprefix/bin/perlcc: Unexpected compiler output
Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.
 Use of uninitialized value $cvname in string eq at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 4981.

 Use of uninitialized value in XS subroutine entry at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Use of uninitialized value in XS subroutine entry at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Invalid version format (non-numeric data) at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/PerlIO/scalar.pm line 4, <__ANONIO__> line 2459.
 Compilation failed in require at /mnt/c/Users/user/wiptlmgr/cperlprefix/lib/5.30.0/x86_64-linux/B/C.pm line 6225, <__ANONIO__> line 2459.
 CHECK failed--call queue aborted.

vadimkantorov commented 4 years ago

It seems that static modules also misbehave in perlcc-produced source code:

Compiling without File/Glob and PerlIO/Scalar leads to following errors while compiling tlmgr.c produced by perlcc. It seems that these boot function names are not formatted well in this case.


tlmgr.c:77:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Digest/MD5 (pTHX_ CV* cv);
                          ^
tlmgr.c:78:24: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_List/Util (pTHX_ CV* cv);
                        ^
tlmgr.c:79:24: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Time/HiRes (pTHX_ CV* cv);
                        ^
tlmgr.c:80:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/Byte (pTHX_ CV* cv);
                          ^
tlmgr.c:81:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/CN (pTHX_ CV* cv);
                          ^
tlmgr.c:82:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/EBCDIC (pTHX_ CV* cv);
                          ^
tlmgr.c:83:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘/’ token
 EXTERN_C void boot_Encode/JP (pTHX_ CV* cv);

tlmgr.c source code: tlmgr.zip

vadimkantorov commented 4 years ago

To summarize the encountered problems:

XSLoader-using modules fail when compiled statically with Use of uninitialized value in XS subroutine entry
XSLoader is used in File/Glob which is used at install-time, leading to failing install if File/Glob is compiled statically
perlcc produces malformed code for static modules init functions
disabling dynamic module loading by -Uusedl fails very badly during cperl compilation

XSLoader seems to be inside DynaLoader.o package which by default is linked into libperl.a.