[help/bug?] dynamic importing of python modules from the file system

[jmrgibson]

We have a test application running pytest that uses some pip packages and some rust code. The test writers develop mainly on the pytest tests, but rarely change the rust code and python packages. To keep iteration time low, I'm bundling the python interpreter, pip packages and rust code using pyoxidizer, but allowing users to run their pytest tests from the file system. Our build system scans the file system for changes, and then determines whether a rebuild of the executable (rust or pip changes), or just a re-run of the tests (pytest code changes) is necessary.

Pytest internally uses the (py)[https://github.com/pytest-dev/py] pacakge to dynamically import the test modules from the file system. It first finds the test file, adds the directory to the path, then uses __import__ to load the module. (see https://github.com/pytest-dev/py/blob/master/py/_path/local.py#L701)

This breaks when using pyoxidizer, as it can't find the module, even if sys.path is correct. I've tried rewriting it to use importlib.load_module but that also fails.

I have locally modified copy of py that uses the code below instead of __import__, but I'm wondering if pyoxidier is intending to disallow the dynamic import of modules from the file system.

spec = importlib.util.spec_from_file_location(modname, self.realpath())
new_mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(new_mod)

[indygreg]

PyOxidizer should not be preventing normal file-based import from working. Although by default it won't register the traditional filesystem importer on sys.meta_path. You need to pass filesystem_importer=True to the PythonInterpreterConfig used to configure the executable. This value should be implied if sys_paths is also specified.

From the context of your PyOxidizer application, what are the contents of sys.meta_path and sys.path? You should see a <class '_frozen_importlib_external.PathFinder'> on sys.meta_path. This is the standard Python filesystem-based importer.

[jmrgibson]

Ah, yeah, after changing filesystem_importer=True this works. It might be worth mentioning that in the FAQ. I'll put up a PR

[jmrgibson]

PR for more explicit docs https://github.com/indygreg/PyOxidizer/pull/227

[jmrgibson]

Hey @indygreg , thanks for your help so far! I now have everything working fine on mac and linux, but I'm having trouble on windows importing modules from the file system with native extensions. Specifically, I have a pre-existing rust crate, built as a cdylib, that uses pyo3 to expose itself as a python module (called pymodule_rs).

Here is the error I'm getting on windows:

Traceback (most recent call last):
  File "_pytest.config", line 466, in _importconftest
  File "py._path.local", line 710, in pyimport
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "D:\jgibson\tests\conftest.py", line 25, in <module>
    import pyrustmodule
  File "D:/jgibson/pymodule_rs\pyrustmodule\__init__.py", line 6, in <module>
    from .pymodule_rs import *
ImportError: DLL load failed: The specified module could not be found.

Here are the tests I've run so far

1. I can run python3.7 (x64) and import pymodule_rs from the REPL without issues
2. I've used depwalker to confirm that all executables are x64
3. depwalker shows that pymodule_rs.pyd is 64 bit, and depends on libpython3.7.dll
4. depwalker shows that the .exe built by pyoxidize is also 64 bit, and doesn't have any other dependencies aside from windows APIs.

Am I going to run into issues dynamically loading libpython, and also having it embedded in the executable? strange that I didn't run into these on other operating systems

[indygreg]

I suspect the most recent issue with a failure to load the .pyd has to do with what Python distribution is being used.

Until the yet-to-be-released PyOxidizer 0.7, the Windows Python distribution being used did not support loading .pyd extension modules. This is due to how the Python distribution was being built and Windows symbol visibility rules. In PyOxidizer 0.7, you can now use default_python_distribution(flavor="standalone_dynamic") to specify you want a Python distribution supporting dynamic linking. Pre-built .pyd extension modules should "just work" with the standard filesystem importers using this Python distribution.

The feature is still being fleshed out. There is definitely room to improve the documentation on extension compatibility. I'll try to do this before the 0.7 release. In the meantime, if you fetch the latest commits from Git and use that, it should do what you want.