JuliaPy / PyCall.jl

Package to call Python functions from the Julia language
MIT License
1.46k stars 187 forks source link

Workaround missing main symbol in Python >= 3.9 #937

Closed tkf closed 2 years ago

tkf commented 2 years ago

This patch changes Libdl.dlsym(proc_handle, :main) to Libdl.dlsym(proc_handle, :_start) to fix https://github.com/JuliaPy/pyjulia/issues/459

In Python >= 3.9, main cannot be accessed through the process handle (dlopen(NULL, ...) ):

$ python3.9 -c 'import ctypes; ctypes.CDLL(None).main'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/arakaki/.pyenv/versions/3.9.7/bin/python3.9: undefined symbol: main

while it has been possible for < 3.9.

$ python3.8 -c 'import ctypes; ctypes.CDLL(None).main'  # no error

I'm not sure why this is the case since python3.9 binary does seem to have main:

$ objdump -dC ~/.pyenv/versions/3.9.7/bin/python3.9 | grep -e '<main>' -A5
0000000000000720 <main>:
 720:   e9 db ff ff ff          jmpq   700 <Py_BytesMain@plt>
 725:   66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
 72c:   00 00 00
 72f:   90                      nop

--
 74d:   48 8d 3d cc ff ff ff    lea    -0x34(%rip),%rdi        # 720 <main>
 754:   ff 15 86 08 20 00       callq  *0x200886(%rip)        # 200fe0 <__libc_start_main@GLIBC_2.2.5>
 75a:   f4                      hlt
 75b:   0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

That said, https://github.com/JuliaPy/pyjulia/issues/459 can now be solved by checking _start instead. (Should we check main first and then fallback to _start?)

A better fix probably is to use Preferences.jl as the build-time option (#835) and get rid of automatic detection. But I wonder if this is good enough for now?

@Keno It'd be great if you can look at this, as you wrote this originally in #419.

codecov-commenter commented 2 years ago

Codecov Report

Merging #937 (b787aba) into master (4e5c12c) will not change coverage. The diff coverage is 0.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #937   +/-   ##
=======================================
  Coverage   68.24%   68.24%           
=======================================
  Files          20       20           
  Lines        2012     2012           
=======================================
  Hits         1373     1373           
  Misses        639      639           
Flag Coverage Δ
unittests 68.24% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/startup.jl 53.12% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4e5c12c...b787aba. Read the comment docs.

Keno commented 2 years ago

Seems reasonable to me. Any symbol should work.

tkf commented 2 years ago

Thanks! By any chance, do you have any idea why this is happening? I'm just puzzled that main is "gone," presumably after it's called via _start.

Keno commented 2 years ago

Did you check the dynamic symbol table nm -D? They may have decided to hide it.

tkf commented 2 years ago

Indeed main does not show up in python3.9 but it does in python3.8!

$ nm -D ~/.pyenv/versions/3.9.7/bin/python3.9
0000000000201010 B __bss_start
                 w __cxa_finalize
0000000000201000 D __data_start
0000000000201000 W data_start
0000000000201010 D _edata
0000000000201018 B _end
00000000000008b4 T _fini
                 w __gmon_start__
00000000000006d0 T _init
00000000000008c0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
00000000000008b0 T __libc_csu_fini
0000000000000840 T __libc_csu_init
                 U __libc_start_main
                 U Py_BytesMain
0000000000000730 T _start
$ nm -D ~/.pyenv/versions/3.8.12/bin/python3.8
0000000000201010 B __bss_start
                 w __cxa_finalize
0000000000201000 D __data_start
0000000000201000 W data_start
0000000000201010 D _edata
0000000000201018 B _end
00000000000008e4 T _fini
                 w __gmon_start__
0000000000000700 T _init
00000000000008f0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
00000000000008e0 T __libc_csu_fini
0000000000000870 T __libc_csu_init
                 U __libc_start_main
0000000000000750 T main
                 U Py_BytesMain
0000000000000760 T _start