Can we build the project without requirements.txt ?

[eci-aashish]

Hello Team,

My requirement to bulid My python project and requrements.txt separately using pex.

As the requirements.txt is too large, so I am looking the way to build my project separately and then run this pex build by just specifying already installed virtual environment i.e. I need to create virtual environment first and then just pass the path of this virtual environment to run the pex build.

Don't want to specify requirements.txt in pex command as its getting heavier.

Please guide.

Thanks

[jsirois]

For example, with this setup 1st party code:

/tmp/example $ tree
.
└── package
    ├── bar.py
    ├── baz.py
    └── foo.py

1 directory, 3 files
/tmp/example $ cat package/foo.py
from package import bar, baz

if __name__ == "__main__":
    print(
        "3rdparty paths:\n{paths}".format(paths="\n".join((bar.colors_path(), baz.cowsay_path())))
    )

/tmp/example $ cat package/bar.py
import colors

def colors_path():
    return colors.__file__

/tmp/example $ cat package/baz.py
import cowsay

def cowsay_path():
    return cowsay.__file__

I can try to run it and it fails since I don't have the 3rdparty requirements:

/tmp/example $ python -m package.foo
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/example/package/foo.py", line 1, in <module>
    from package import bar, baz
  File "/tmp/example/package/bar.py", line 1, in <module>
    import colors
ModuleNotFoundError: No module named 'colors'

But, I can create a 3rdparty dependency PEX and use it like a python interpreter:

/tmp/example $ pex ansicolors cowsay -o 3rdparty.pex
/tmp/example $ ./3rdparty.pex -m package.foo
3rdparty paths:
/home/jsirois/.pex/installed_wheels/00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__init__.py
/home/jsirois/.pex/installed_wheels/274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a/cowsay-6.1-py3-none-any.whl/cowsay/__init__.py

So, here, just as with a normal python interpreter, the $PWD is on the sys.path and the PEX can see both the 1st party code in PWD and the 3rdparty dependencies in the PEX file and everything works.

You can also explicitly steer the PEX to find the 1st party code:

# Can't find it.
/tmp/example $ mv 3rdparty.pex ..
/tmp $ ./3rdparty.pex -m package.foo
...
  File "/home/jsirois/.pex/unzipped_pexes/c144d47a2d979a2605964d3858d98aaf6adf9f6f/.bootstrap/pex/pex.py", line 818, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "<frozen runpy>", line 222, in run_module
  File "<frozen runpy>", line 140, in _get_module_details
ImportError: Error while finding module specification for 'package.foo' (ModuleNotFoundError: No module named 'package')

# Still can't find it because PEXes are hermetic by default and ignore your environment (the `PYTHONPATH` setting in this case).
$ PYTHONPATH=example ./3rdparty.pex -m package.foo
  File "/home/jsirois/.pex/unzipped_pexes/c144d47a2d979a2605964d3858d98aaf6adf9f6f/.bootstrap/pex/pex.py", line 818, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "<frozen runpy>", line 222, in run_module
  File "<frozen runpy>", line 140, in _get_module_details
ImportError: Error while finding module specification for 'package.foo' (ModuleNotFoundError: No module named 'package')

# Now can find it.
$ PEX_INHERIT_PATH=prefer PYTHONPATH=example ./3rdparty.pex -m package.foo
3rdparty paths:
/home/jsirois/.pex/installed_wheels/00d2dde5a675579325902536738dd27e4fac1fd68f773fe36c21044eb559e187/ansicolors-1.1.8-py2.py3-none-any.whl/colors/__init__.py
/home/jsirois/.pex/installed_wheels/274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a/cowsay-6.1-py3-none-any.whl/cowsay/__init__.py

In short there are many ways to separate 1st and 3rdparty code in PEXes.

You should read up on both the output of pex --help and pex --help-variables now that you know about the "inherit path" option. Another similar runtime option is PEX_EXTRA_SYS_PATH.

[jsirois]

@eci-aashish I've labelled this as a question and would like to close as answered if you can confirm either the --inherit-path PEX build-time option or PEX_INHERIT_PATH runtime equivalent help you out.

[jsirois]

@eci-aashish I gave you a good amount of detail there to work with. I'm going to assume the silence means your question is answered and close. Speak up if not.