Closed hakonhagland closed 3 years ago
niner
commented
3 years ago This is very odd. Those 0xDD seem to appear in many places in that output (like 06dict.t) but the whole distribution doesn't contain a single one of them:
inline-python-pm (master=)> find . -type f | grep -v '\.git' | xargs hexdump -e '8/1 "%02x_L[red:0xDD] " "\n"' | grep ddCan you try a different user or machine? Maybe it's something in your environment.
hakonhagland
commented
3 years ago Yes this works fine on a fresh Ubuntu 20.04 VM:
$ git clone https://github.com/niner/inline-python-pm.git
$ cd inline-python-pm
$ perl --version
This is perl 5, version 30, subversion 0 (v5.30.0) built for x86_64-linux-gnu-thread-multi
$ python3 --version
Python 3.8.2
$ sudo apt-get update
$ sudo apt-get install python3-dev cpanminus
$ sudo cpanm --installdeps .
$ sudo cpanm Parse::RecDescent
$ perl Makefile.PL
Found these python executables on your PATH:
1. /usr/bin/python3
2. /usr/bin/python
Using the only python 3 executable I could find even though python 2 was also found
Set the INLINE_PYTHON_EXECUTABLE environment variable to the full path to your python executable to override this selection.
Using /usr/bin/python3
Using These Settings:
Extra Libs: -lcrypt -lpthread -ldl -lutil -lm
Python Lib: -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -lpython3.8
Includes: -I/usr/include/python3.8
Extra Flags: -DPY_MAJOR_VERSION=3
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Inline::Python
Writing MYMETA.yml and MYMETA.json
$ make
cp Python.pm blib/lib/Inline/Python.pm
cp Python.pod blib/lib/Inline/Python.pod
Running Mkbootstrap for Python ()
chmod 644 "Python.bs"
"/usr/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- Python.bs blib/arch/auto/Inline/Python/Python.bs 644
"/usr/bin/perl" "/usr/share/perl/5.30/ExtUtils/xsubpp" -typemap '/usr/share/perl/5.30/ExtUtils/typemap' Python.xs > Python.xsc
mv Python.xsc Python.c
x86_64-linux-gnu-gcc -c -I/usr/include/python3.8 -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"0.56\" -DXS_VERSION=\"0.56\" -fPIC "-I/usr/lib/x86_64-linux-gnu/perl/5.30/CORE" -DEXPOSE_PERL -DCREATE_PYTHON -UCREATE_PERL Python.c
Python.xs: In function ‘do_pyinit’:
Python.xs:52:5: warning: implicit declaration of function ‘initperl’ [-Wimplicit-function-declaration]
52 | initperl();
| ^~~~~~~~
x86_64-linux-gnu-gcc -c -I/usr/include/python3.8 -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"0.56\" -DXS_VERSION=\"0.56\" -fPIC "-I/usr/lib/x86_64-linux-gnu/perl/5.30/CORE" -DEXPOSE_PERL -DCREATE_PYTHON -UCREATE_PERL py2pl.c
py2pl.c: In function ‘croak_python_exception’:
py2pl.c:571:36: warning: format ‘%i’ expects argument of type ‘int’, but argument 4 has type ‘long int’ [-Wformat=]
571 | croak("%s: %s at line %i\n", ((PyTypeObject *)ex_type)->tp_name, c_ex_message, PyInt_AsLong(tb_lineno));
| ~^
| |
| int
| %li
x86_64-linux-gnu-gcc -c -I/usr/include/python3.8 -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"0.56\" -DXS_VERSION=\"0.56\" -fPIC "-I/usr/lib/x86_64-linux-gnu/perl/5.30/CORE" -DEXPOSE_PERL -DCREATE_PYTHON -UCREATE_PERL perlmodule.c
x86_64-linux-gnu-gcc -c -I/usr/include/python3.8 -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"0.56\" -DXS_VERSION=\"0.56\" -fPIC "-I/usr/lib/x86_64-linux-gnu/perl/5.30/CORE" -DEXPOSE_PERL -DCREATE_PYTHON -UCREATE_PERL util.c
rm -f blib/arch/auto/Inline/Python/Python.so
LD_RUN_PATH="/usr/lib/python3.8/config-3.8-x86_64-linux-gnu" x86_64-linux-gnu-gcc -shared -L/usr/local/lib -fstack-protector-strong Python.o py2pl.o perlmodule.o util.o -o blib/arch/auto/Inline/Python/Python.so \
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -lpython3.8 -lcrypt -lpthread -ldl -lutil -lm \
chmod 755 blib/arch/auto/Inline/Python/Python.so
Manifying 1 pod document
$ make test
"/usr/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- Python.bs blib/arch/auto/Inline/Python/Python.bs 644
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00init.t ................ ok
t/01testpl.t .............. ok
t/02testpl.t .............. ok
t/03parse.t ............... ok
t/04func.t ................ ok
t/05JAxH.t ................ ok
t/06dict.t ................ ok
t/07nherit.t .............. ok
t/08ipyobj.t .............. ok
t/09bind.t ................ ok
t/10pyeval.t .............. ok
t/11factor.t .............. ok
t/12evnodd.t .............. ok
t/13fibbon.t .............. ok
t/14study.t ............... ok
t/15anon.t ................ ok
t/16evalpy.t .............. ok
t/17once.t ................ ok
t/18newclass.t ............ ok
t/19testref.t ............. ok
t/20unicode.t ............. ok
t/21arrayref.t ............ ok
t/22int.t ................. ok
t/23getattr.t ............. ok
t/24getitem.t ............. ok
t/25py_sub.t .............. ok
t/26undef.t ............... ok
t/27pyattrs.t ............. ok
t/28exception.t ........... ok
t/29named_params.t ........ ok
t/30floats.t .............. ok
t/31stringification.t ..... ok
t/32boolean.t ............. ok
t/33reinit.t .............. ok
t/34env.t ................. ok
t/35dictunicodememleak.t .. ok
t/36utfstring.t ........... ok
t/cmp.t ................... ok
All tests successful.
Files=38, Tests=842, 7 wallclock secs ( 0.10 usr 0.04 sys + 6.43 cusr 0.47 csys = 7.04 CPU)
Result: PASSOk, then it really is no issue with Inline::Python but some weirdness in the environment that manages to mightily confuse Perl. Closing.
hakonhagland
commented
3 years ago Hi. I investigated this a little bit more. I think it could be related to that I use pyenv. It seems that the problem is that the generated shared object Python.so is linked statically with libpython3.9d.a instead of dynamically with libpython3.9d.so when I use the pyenv python. I notice from the output of make that
if I use the pyenv installed Python 3.9.3, Python.so is generated by the command:
cc -shared -O2 -L/usr/local/lib -fstack-protector-strong Python.o py2pl.o perlmodule.o util.o \
-o blib/arch/auto/Inline/Python/Python.so \
-L/home/hakon/.pyenv/versions/3.9.3-debug/lib/python3.9/config-3.9d-x86_64-linux-gnu \
-lpython3.9d -lcrypt -lpthread -ldl -lutil -lmIf I instead use the system python 3.9.4, Python.so is generated using:
LD_RUN_PATH="/usr/lib/python3.9/config-3.9-x86_64-linux-gnu" cc -shared -O2 -L/usr/local/lib \
-fstack-protector-strong Python.o py2pl.o perlmodule.o util.o -o blib/arch/auto/Inline/Python/Python.so \
-L/usr/lib/python3.9/config-3.9-x86_64-linux-gnu -lpython3.9 -lcrypt -lpthread -ldl -lutil -lm Now, there is an interesting difference in the above to library paths:
-L/home/hakon/.pyenv/versions/3.9.3-debug/lib/python3.9/config-3.9d-x86_64-linux-gnu, and-L/usr/lib/python3.9/config-3.9-x86_64-linux-gnuIn the first case there is no shared object libpython3.9d.so in the given path, but for the second case (the system python), there is a shared object /usr/lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.so in the path.
I think that in the first case, since the .so is missing, the cc command is using the static library /home/hakon/.pyenv/versions/3.9.3-debug/lib/python3.9/config-3.9d-x86_64-linux-gnu/libpython3.9d.a instead.
I checked Makefile.PL, and I see that on line 154 it is using the config variable LIBPL to locate the directory of libpython.so. However, there is another config variable called LIBDIR, see this Q&A, which works correctly for me. Compare
for the pyenv python:
$ python3 -c "from distutils import sysconfig; print('LIBDIR={}'.format(sysconfig.get_config_var('LIBDIR'))); print('LIBPL={}'.format(sysconfig.get_config_var('LIBPL')))"
LIBDIR=/home/hakon/.pyenv/versions/3.9.3-debug/lib
LIBPL=/home/hakon/.pyenv/versions/3.9.3-debug/lib/python3.9/config-3.9d-x86_64-linux-gnuand for the system python:
$ python3 -c "from distutils import sysconfig; print('LIBDIR={}'.format(sysconfig.get_config_var('LIBDIR'))); print('LIBPL={}'.format(sysconfig.get_config_var('LIBPL')))"
LIBDIR=/usr/lib/x86_64-linux-gnu
LIBPL=/usr/lib/python3.9/config-3.9-x86_64-linux-gnuIn both cases above, the shared object exists in LIBDIR. Maybe Makefile.PL needs to be patched to use LIBDIR instead of LIBPL ?
Commands to reproduce:
I choose to focus on the first test failure in this issue:
By adding debug print statement in
14study.tI can see thatpy_study_packageon line 16 returns the following hash:Where the key of the sub hash
classescan be printed usingprintf "%vX\n", $key. This shows that the key contains the bytes: