ModuleNotFoundError: No module named "models"

[victoris93]

Hi everyone,

I'm wondering if there's an elegant solution to the error ModuleNotFound not involving sys.path.append(). The situation is the following:

1. Script train_model.py is in /work.
2. Jobs are sent via submitit from train_model.py. AutoEncoder is imported from models in the beginning of train_model.py. Module models.py is also in /work.
3. The output folder for submitit jobs is/work/outputs/date/time. Job pickles are stored there.

My intuition is that the jobs sent from within train_model.py and the job on which train_model.py is running do not share the same PYTHONPATH (the env should be fine because this problem does not extend on installed packages). What would be your recommendation of an optimal solution to this one (if any)?

[baldassarreFe]

There isn't enough info to provide a solution. What is the folder structure? Are the imports relative or absolute? What python environment are you using (venv, conda, ...)? Are you setting the PYTHONPATH manually before the script?

These are some steps to troubleshoot:

Log the PYTHONPATH in your script, both in the script that submits the job and in the script that runs the job (you'll need to log it before the failing imports).

Is your PYTHONPATH set?

• No -> set it before submitting the jobs
• Yes -> is it using relative paths? make them absolute

In the same places where you log the PYTHONPATH, log also PWD, all CONDA_* and all SUBMITIT_* variables., just in case

[victoris93]

Thanks for getting back to me. I'm using venv; PYTHONPATH is set and it is absolute. I solved the problem by compiling all my modules in a package so the editable installation is in the env. However it seems like a rather time-consuming option.

[baldassarreFe]

Great that you managed to find a workaround. In my experience, installing your scripts as an editable package should not be necessary and something this should work:

conda activate xxx  # or your venv
export PYTHONPATH="${HOME}/projects/bla:${PYHONPATH}"
python "${HOME}/projects/bla/run_with_submitit.py"

Then in ${HOME}/projects/bla you may have main.py that imports other files like this:

import model  # ~/projects/bla/model.py
import utils.somthing  # ~/projects/bla/utils/something.py

def main():
  ...

[victoris93]

Ok I see, this totally makes sense. Thanks!