indygreg / PyOxidizer

A modern Python application packaging and distribution tool
Mozilla Public License 2.0
5.46k stars 236 forks source link

Shared lib builder drops package directory #169

Open jayvdb opened 4 years ago

jayvdb commented 4 years ago

Shared libs that are in packages that also contain python code are stored in site-packages/foo/bar.so

PyOxidizer is dropping the foo package name, creating conflicts.

A very prominent example is markupsafe, simplejson and logbook all have a _speedups.so

>>> import simplejson
    import simplejson as json
  File "simplejson", line 115, in <module>
  File "simplejson.encoder", line 15, in <module>
  File "simplejson.encoder", line 12, in _import_speedups
AttributeError: module 'markupsafe._speedups' has no attribute 'encode_basestring_ascii'

tornado and websockets and fastparquet all have a speedups.so

cytoolz and h5py and espressomd all have utils.so, and further nested are zmq/backend/cython/utils.so and yt/analysis_modules/photon_simulator/utils.so

dulwich and h5py have _objects.so

xxhash and lmdb have cpython.so

bsdiff4 and espeak have core.so

astropy and sunpy have _compiler.so

gevent and kivy both have _event.so

PyQt4 and PyQt5 have an almost identical set of built modules under their respective directories.

Cryptodome and Crypto also share a set of similar names.

Third level objects sklearn/manifold/_utils.so and sklearn/tree/_utils.so probably also have this problem, breaking sklearn even if only it is installed.

Third level objects gssapi/raw/misc.so and samba/dcerpc/misc.so likely also conflict.

Third level astropy/stats/_stats.so and scipy/stats/_stats.so

As noted at https://github.com/indygreg/PyOxidizer/issues/167 , brotli and brotlipy both expect from brotli import _brotli to import their .so , but that example is a bit messy as the two libraries conflict anyway.

And that is just what I can find in my site-packages :P Obviously there will be many more conflicts I am unaware of.

jayvdb commented 4 years ago

At least one of the problem is they both have symbols PyInit__speedups, and both are written into pyoxidizer/config.c, so probably one is executed twice.