google / subpar

Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.
Apache License 2.0
567 stars 69 forks source link

Conflicting package names #116

Open wrobstory opened 5 years ago

wrobstory commented 5 years ago

Hi subpar folks! Apologies if this is a duplicate but I couldn't find any prior art.

When using tensorflow in subpar, in the import roots you end up with a ton of packages named google:

$ find unzipped_par/ -name google -type d | sort
unzipped_par/pypi2__google_api_core_1_14_2/google
unzipped_par/pypi2__googleapis_common_protos_1_6_0/google
unzipped_par/pypi2__google_auth_1_6_3/google
unzipped_par/pypi2__google_cloud_bigquery_1_6_1/google
unzipped_par/pypi2__google_cloud_bigtable_0_31_1/google
unzipped_par/pypi2__google_cloud_core_0_28_1/google
unzipped_par/pypi2__google_cloud_datastore_1_7_4/google
unzipped_par/pypi2__google_cloud_pubsub_0_39_0/google
unzipped_par/pypi2__google_resumable_media_0_3_2/google
unzipped_par/pypi2__grpc_google_iam_v1_0_11_4/google
unzipped_par/pypi2__protobuf_3_7_0/google
unzipped_par/pypi2__proto_google_cloud_datastore_v1_0_90_4/google
unzipped_par/pypi2__tensorflow_1_14_0/tensorflow/include/external/protobuf_archive/python/google
unzipped_par/pypi2__tensorflow_1_14_0/tensorflow/include/external/protobuf_archive/src/google
unzipped_par/pypi2__tensorflow_1_14_0/tensorflow/include/google

When you import, it resolves to the first import root:

(Pdb) google.__path__
['/pay/home/df/dev-df/unzipped_par/pypi2__google_api_core_1_14_2/google']

I believe that in a virtualenv-like environment, everything would get resolved under a single google package with subpackages for each library:

$ ls import_test_virtualenv/lib/python2.7/site-packages/google
api  api_core  auth  logging  longrunning  oauth2  protobuf  rpc  type

I'm trying to figure out if I'm missing something: should subpar be resolving these separate packages in a similar way so that we don't have a first-import-wins situation?

wrobstory commented 5 years ago

Left out an important detail: when you try to import from a library like protobuf, you get the following error:

from google.protobuf.json_format import ParseDict
ImportError: No module named protobuf.json_format