criteo / cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Apache License 2.0
45 stars 21 forks source link

Handle the large_pex case when running from an unzipped pex #120

Closed jcuquemelle closed 6 months ago

jcuquemelle commented 6 months ago

This happens when we run python code from an unzipped large pex, and the code that is run will itself try and rebuild a large (zipped) pex (e.g. by launching a spark job that will call cluster_pack.upload_env)

The detect_archive_name function must be aware that it is currently running from an unzipped pex in order to correctly retrieve the original zipped pex name