idmlaser developer not compiling update_ages.cpp

[krosenfeld-IDM]

on jb_modulify/ #[19bba72] I am following the developer version for installing idmlaser from https://github.com/InstituteforDiseaseModeling/laser/wiki/Using-idmlaser-as-a-pip-installable-module. It does not appear to be compiling update_ages.cpp.

My commands (from within the jb/ directory):

pip install -e .
python3 -m idmlaser.utils.build_template_workspace

I build the CCS example in jb/sandbox and then get the error:

(/home/krosenfeld/projects/202407_laser/env) krosenfeld@internal.idm.ctr@ipapvwks23:~/projects/202407_laser/jb/sandbox$ python -m idmlaser.measles
Traceback (most recent call last):
  File "/home/krosenfeld/projects/202407_laser/env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/krosenfeld/projects/202407_laser/env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/krosenfeld/projects/202407_laser/jb/src/idmlaser/measles.py", line 17, in <module>
    from . import sir_numpy_c as model
  File "/home/krosenfeld/projects/202407_laser/jb/src/idmlaser/sir_numpy_c.py", line 76, in <module>
    update_ages_lib = ctypes.CDLL(shared_lib_path)
  File "/home/krosenfeld/projects/202407_laser/env/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/krosenfeld/projects/202407_laser/jb/src/idmlaser/update_ages.so: cannot open shared object file: No such file or directory

[krosenfeld-IDM]

Before switching ot this branch I was compiling with make update_ages.so but this isn't working:

(/home/krosenfeld/projects/202407_laser/env) krosenfeld@internal.idm.ctr@ipapvwks23:~/projects/202407_laser/jb/sandbox$ make update_ages.so
make: *** No rule to make target 'update_ages.cpp', needed by 'update_ages.so'.  Stop.

[krosenfeld-IDM]

Looking at my sandbox/ directory, there was no update_ages.cpp installed:

-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr  310 Jul 15 04:28 demographics_settings.py
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.2K Jul 15 04:28 eula_binned.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 106K Jul 15 04:28 eula_pops_20yr_noaging.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr  360 Jul 15 04:28 fits.npy
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.7K Jul 15 04:44 makefile
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 6.5M Jul 15 04:28 modeled_pop.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.8M Jul 15 04:28 modeled_pop.csv.gz
drwxrwxr-x 2 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 4.0K Jul 15 04:28 __pycache__
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 458K Jul 15 04:28 QuickStart.ipynb
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.4K Jul 15 04:28 settings.py

[krosenfeld-IDM]

I had tried to install idmlaser by running pip install -e . from within the jb/ directory

[jonathanhhb]

Harumph. No repro for me on quick test. Will recheck first thing in morning.

[krosenfeld-IDM]

Appreciated. All I should need is the jb/ directory, right? Do I need to use the idm pypi in the index-url for the pip install -e .? It seemed to be workign withough it.

[krosenfeld-IDM]

I guess it make sense that there is no update_ages.cpp in the sandbox/ but why is there a makefile?

[jonathanhhb]

Appreciated. All I should need is the jb/ directory, right? Do I need to use the idm pypi in the index-url for the pip install -e .? It seemed to be workign withough it.

Ya, just pip install -e . should do it.

[jonathanhhb]

I guess it make sense that there is no update_ages.cpp in the sandbox/ but why is there a makefile?

I think it was for if you wanted to change your demographics_settings.py (population, number of nodes, for CCS) and get new input files.

update_ages.cpp should be compiled as part of the install.

[jonathanhhb]

The files you need to copy from a sandbox into your COMPS/idmtools inputs folder should be well captured by the contents of inputs or inputs_ew here:

├── inputs
│   ├── demographics_settings.py
│   ├── fits.npy
│   ├── modeled_pop.csv.gz
│   └── settings.py
├── inputs_ew
│   ├── attraction_probabilities.csv
│   ├── cbrs_ew.csv
│   ├── cities.csv
│   ├── demographics_settings.py
│   ├── engwal_modeled.csv.gz
│   ├── fits.npy
│   └── settings.py
├── laser.def
├── laser.id
├── run_laser_with_sif.py
├── sifs
│   └── laser.sif

[jonathanhhb]

Looking at my sandbox/ directory, there was no update_ages.cpp installed:

-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr  310 Jul 15 04:28 demographics_settings.py
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.2K Jul 15 04:28 eula_binned.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 106K Jul 15 04:28 eula_pops_20yr_noaging.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr  360 Jul 15 04:28 fits.npy
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.7K Jul 15 04:44 makefile
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 6.5M Jul 15 04:28 modeled_pop.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.8M Jul 15 04:28 modeled_pop.csv.gz
drwxrwxr-x 2 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 4.0K Jul 15 04:28 __pycache__
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 458K Jul 15 04:28 QuickStart.ipynb
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.4K Jul 15 04:28 settings.py

You only need data files from the sandbox now. All code is in the installed package.

[krosenfeld-IDM]

adding notes to issue thread for continuity:

• it seemed that the make command was not running the generate the update_ages.so. Once I added that step by hand I was able to pip install

Relevant code below that appears to to be running?

https://github.com/InstituteforDiseaseModeling/laser/blob/f598b877f2b20694a3aaa60ef6bd3a7171eab047/jb/setup.py#L14-L26

[jonathanhhb]

Line 20 should do the equivalent of what the make command was doing to build the cpp code into a .so during the installation.

Are you install from a .whl (wheel) in your .def?

[krosenfeld-IDM]

I'm not installing from a .whl, I'm dong a git clone and pip install: https://github.com/krosenfeld-IDM/202407_laser/blob/16a46de7b4fad5187ef0371e97cb833b99d31dbd/jb_jenkins/Dockerfile#L16

[jonathanhhb]

@krosenfeld-IDM Should I see if I can find out why update_ages.cpp wasn't getting compiled during install?

[krosenfeld-IDM]

Thanks, I'm still seeing the error.

It would be good to fix if continuing development / use from jb_modulify. Maybe we can discuss on Wednesday what you, @KevinMcCarthyAtIDM , and @clorton 's thoughts are for next steps?

You can try reproducing it by building a clean codespace and then:

cd jb
 pip install -e .
 cd ..
 mkdir workspace
 cd workspace/
 python -m idmlaser.utils.build_template_workspace
# building in ./EW
 cd EW
 python -m idmlaser.measles

produces:

Traceback (most recent call last):
  File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspaces/laser/jb/src/idmlaser/measles.py", line 17, in <module>
    from . import sir_numpy_c as model
  File "/workspaces/laser/jb/src/idmlaser/sir_numpy_c.py", line 76, in <module>
    update_ages_lib = ctypes.CDLL(shared_lib_path)
  File "/usr/local/python/3.10.13/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /workspaces/laser/jb/src/idmlaser/update_ages.so: cannot open shared object file: No such file or directory

[jonathanhhb]

Thanks, I'm still seeing the error.

It would be good to fix if continuing development / use from jb_modulify. Maybe we can discuss on Wednesday what you, @KevinMcCarthyAtIDM , and @clorton 's thoughts are for next steps?

You can try reproducing it by building a clean codespace and then:

cd jb
 pip install -e .
 cd ..
 mkdir workspace
 cd workspace/
 python -m idmlaser.utils.build_template_workspace
# building in ./EW
 cd EW
 python -m idmlaser.measles

produces:

Traceback (most recent call last):
  File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspaces/laser/jb/src/idmlaser/measles.py", line 17, in <module>
    from . import sir_numpy_c as model
  File "/workspaces/laser/jb/src/idmlaser/sir_numpy_c.py", line 76, in <module>
    update_ages_lib = ctypes.CDLL(shared_lib_path)
  File "/usr/local/python/3.10.13/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /workspaces/laser/jb/src/idmlaser/update_ages.so: cannot open shared object file: No such file or directory

Thanks. Able to repro now. Also I get an error when injecting infections (import cases) with a dev install, so that's worth looking at. This whole thing with compiled C code in a module is a bit new and I want to make sure we're catching and fixing issues.

[jonathanhhb]

Lots of good suggestions from GPT but dev build isn't invoking custom build commands to create .so yet...

[jonathanhhb]

Personally I don't do dev installs. I build a .whl and pip install (from file). I'll keep working on the dev install with Custom Commands for compiling C but the build-and-install-wheel is a completely viable "workaround" if you don't mind the two steps.

[krosenfeld-IDM]

If I use the .whl does that mean everytime I make a change o the python code I have to build the .whl and pip install from the file?

I find it really useful to be able to make a change to the code and immediately jump into the e.g. running the scrip or the debugger from vscode.

[jonathanhhb]

Ya. Sounds like it would be disruptive to your workflow.

[krosenfeld-IDM]

Yeah - I think so. Thanks

[jonathanhhb]

Fixed with: https://github.com/InstituteforDiseaseModeling/laser/commit/dd5c258248b0ed4a01c3f2a7f4df53d8e06b52ec

@krosenfeld-IDM you can resolve or close this if no further work is neded.

[krosenfeld-IDM]

So for the dev install you can either:

cd jb/
pip install -e .
cd src/idmlaser
make update_ages.so

or

python setup.py develop