Problem with mitosis recognizing config module

[yb6599]

Hey @Jacob-Stevens-Haas,

I had some environment issues and had to remove my repo (kalman-sindy-pde) and clone it again. I know I haven't made any changes that would affect the experiments. However, when I run it, mitosis does not recognize the ksindy_pde_figs module, which contains the config.py. The pyproject.toml is still the same,

[tool.mitosis.steps]
pde_grid = ["gen_experiments.gridsearch:run", "ksindy_pde_figs.config:lookup_dict"]

This has been unchanged and I ran experiments this morning, so it must be something to do with the environment setup I did after recloning. The error I get is

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 2
      1 step_0 = mitosis.unpack('gen_experiments.gridsearch:run')
----> 2 lookup_0 = mitosis.unpack('ksindy_pde_figs.config:lookup_dict')
      3 resolved_args_0 = {}
      4 logger = logging.getLogger('gen_experiments.gridsearch')

File ~/Kalman-SINDy-PDE/.venv/lib/python3.10/site-packages/mitosis/__init__.py:597, in unpack(obj_ref)
    595 def unpack(obj_ref: str) -> Any:
    596     modname, _, qualname = obj_ref.partition(":")
--> 597     obj = import_module(modname)
    598     for attr in qualname.split("."):
    599         obj = getattr(obj, attr)

File /usr/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:992, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1004, in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'ksindy_pde_figs'

I haven't been able to figure it out, what should I do?

[yb6599]

Also, running

import mitosis
lookup = mitosis.unpack('ksindy_pde_figs.config:lookup_dict')
print(lookup)

returns the lookup_dict and works as intended.

[Jacob-Stevens-Haas]

:thinking: I wonder... are you sure these were both run in the same environment?

The other thing is... try activating the environment but running

import mitosis
lookup = mitosis.unpack('ksindy_pde_figs.config:lookup_dict')
print(lookup)

from outside the repository. It could be picking up the import based upon sys.path[0]

[yb6599]

I wonder... are you sure these were both run in the same environment?

Yes

try activating the environment but running from outside the repository.

Okay, this gives me:

Traceback (most recent call last):
  File "/home/yash6599/debug.py", line 2, in <module>
    lookup = mitosis.unpack('ksindy_pde_figs.config:lookup_dict')
  File "/home/yash6599/Kalman-SINDy-PDE/.venv/lib/python3.10/site-packages/mitosis/__init__.py", line 600, in unpack
    obj = import_module(modname)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ksindy_pde_figs'

[yb6599]

Update: I've solved this bug, experiments running like before again, the error originated was

pip install -e .
Obtaining file:///home/yash6599/Kalman-SINDy-PDE
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error

  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      error: Multiple top-level packages discovered in a flat-layout: ['trials', 'ksindy_pde_figs'].

      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.

      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:

      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names

      To find more information, look for "package discovery" on setuptools docs.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I've fixed this by setting up setuptools package discovery in pyproject.toml, but is it better if I make src and keep ksindy_pde_figs and trials in src?

[Jacob-Stevens-Haas]

Yeah, well done! that was what I was thinking about with trying to run from outside the folder. I don't know why setuptools is finding the trials folder if it doesn't have an __init__.py. I prefer option 3, setting the packages in pyproject.toml.

Regardless, if you had gone the src route, you wouldn't also move trials into src, because it's not the package source code.

[yb6599]

Got it, thanks!