search

pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.53k stars 259 forks source link

How to build pex file with requirements.txt ? #2377

Closed eci-aashish closed 7 months ago

eci-aashish commented 7 months ago

Hello Team,

I am building my project using pex but getting memory error due to the large size of requirements.txt file.

Increased the memory but still getting below error -

No main module found. This is likely due to the zip requiring ZIP64 extensions due to size or the number of file entries or both. You can work around this limitation in Python's zipimport module by re-building the PEX with --layout packed or --layout loose.

Tried --layout=packed and --layout=loose but still unable to run the pex build file.

Searched for the solution but unable to get any proper solution for this issue.

Please guide me to resolve it.

Thanks

jsirois commented 7 months ago

One thing to note @eci-aashish is that an issue filed without a concrete example is always a bad footing to start from. If you can provide full examples that allow re-producing the issue, that is best.

I am building my project using pex but getting memory error due to the large size of requirements.txt file.

It's not a memory error. Its an error because the zip requires Zip64 extensions and the Python zipimporter does not support 64 bit zips (that's what the error message tries to convey).

Tried --layout=packed and --layout=loose but still unable to run the pex build file.

There are many distributions that do not work properly with complicated sys.path setups; so you should try --venv which will cause the PEX to turn into a venv at runtime with a ~single sys.path site-packages entry that those distributions expect.

There are other distributions that do not tolerate the tricks Pex uses when laying out a venv to save space on your machine (it uses symlinks by default); so you want to use --venv-site-packages-copies.

All together, that means --layout packed --venv --venv-site-packages-copies.

jsirois commented 7 months ago

@eci-aashish I've labelled this as a question. If my suggestion works for you, please let me know and I'll close the question as answered.

jsirois commented 7 months ago

@eci-aashish I gave you the options you were likely missing in your pex command line (--layout packed --venv --venv-site-packages-copies). I'm going to assume the silence means your question is answered and close. Speak up if not.