go-python / gopy

gopy generates a CPython extension module from a go package.
BSD 3-Clause "New" or "Revised" License
2.05k stars 113 forks source link

Output python extension should use py3+ ABI version tagged naming conventions (PEP 3149) #249

Closed justinfx closed 3 years ago

justinfx commented 3 years ago

Given the following build command:

gopy build -name=goplugin ...

A python extension will be compiled with the name _goplugin.so. This is normal for python2, and may be fine for python3 if you never need to build the extension for any other minor versions of python3. But in order to support multiple build variants in a single directory, python3+ builds should make use of ABI version tagged naming: https://www.python.org/dev/peps/pep-3149/

# python2.7
python -c 'from distutils.sysconfig import get_config_var; print(get_config_var("EXT_SUFFIX"))'
#

# python3.7
python -c 'from distutils.sysconfig import get_config_var; print(get_config_var("EXT_SUFFIX"))'
# .cpython-37m-x86_64-linux-gnu.so

# or python3.7
python3-config --extension-suffix
# .cpython-37m-x86_64-linux-gnu.so

The build process that compiles the python extension should be updated to produce something like this:

_goplugin.cpython-37m-x86_64-linux-gnu.so

And then it should be safe to build multiple times into the same target output directory, given that the Go shared library and the generated python source files should be the same either way.

justinfx commented 3 years ago

I am happy to take assignment of this and work on a PR

justinfx commented 3 years ago

After looking at this a bit more, I realized that the current order of operations causes not only the python extension to be linked against libpython, but also the go c-shared library is also linked against python.

$ gopy build -name=goplugin ...

goimports -w goplugin.go
go build -buildmode=c-shared -o goplugin_go.so
python3 build.py
go run patch-leaks.go goplugin.c
go env CC
gcc goplugin.c goplugin_go.so -o _goplugin.so \
    -I/path/to/python/include/python3.7 \
    -L/path/to/python3.7/lib \
    -lpython3.7m ...

This produces:

.cpython-37m-x86_64-linux-gnu.so     # links against goplugin_go.so and libpython
goplugin_go.so                                    # links against libpython

Would it make more sense to change the order in which the build commands run so that:

  1. Go source is generated
  2. C source is generated
  3. patch memory leak run
  4. Build a single shared library with the Go build tool, which picks up both the go and c sources and links
$ gopy build -name=goplugin ...

goimports -w goplugin.go
python3 build.py
go run patch-leaks.go goplugin.c
export CGO_CFLAGS='-I/path/to/python/include/python3.7 -fPIC -0fast ...'
export CGO_LDFLAGS='-L/path/to/python3.7/lib -lpython3.7m ...'
go build -buildmode=c-shared -o _goplugin.cpython-37m-x86_64-linux-gnu.so

Are there any reason why gopy should keep producing two shared libraries and link between them and both to libpython? If we only produce a single shared library this would also negate the need for #250 Really we still can't put everything in the same directory for multiple minor versions of libpython without the go shared library also clashing.