search

FAIR-Chem / fairchem

FAIR Chemistry's library of machine learning methods for chemistry
https://opencatalystproject.org/
Other
774 stars 243 forks source link

Try removing importlib - breaks multiprocess pickling #746

Closed rayg1234 closed 3 months ago

rayg1234 commented 3 months ago

importlib is breaking multiprocess pickling.

pytest locally works for tests that uses python multiprocess: eg: pytest tests/core/common/test_gp_utils.py (from eq2_graph_parallel branch)

but on CI, which uses ./packages/fairchem-core/pyproject.toml was breaking with errors like:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/home/rgao/micromamba/envs/fair-chem/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/rgao/micromamba/envs/fair-chem/lib/python3.11/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'private'

I traced this back to the option --import-mode importlib in the pytest tool.pytest.ini_options config. This somehow prevents pickling python modules correctly. I tried disabling this seems have no effect on tests and creates more consistency between local development and CI.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark: