Closed krosenfeld-IDM closed 1 month ago
Before switching ot this branch I was compiling with make update_ages.so
but this isn't working:
(/home/krosenfeld/projects/202407_laser/env) krosenfeld@internal.idm.ctr@ipapvwks23:~/projects/202407_laser/jb/sandbox$ make update_ages.so
make: *** No rule to make target 'update_ages.cpp', needed by 'update_ages.so'. Stop.
Looking at my sandbox/
directory, there was no update_ages.cpp
installed:
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 310 Jul 15 04:28 demographics_settings.py
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.2K Jul 15 04:28 eula_binned.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 106K Jul 15 04:28 eula_pops_20yr_noaging.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 360 Jul 15 04:28 fits.npy
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.7K Jul 15 04:44 makefile
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 6.5M Jul 15 04:28 modeled_pop.csv
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.8M Jul 15 04:28 modeled_pop.csv.gz
drwxrwxr-x 2 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 4.0K Jul 15 04:28 __pycache__
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 458K Jul 15 04:28 QuickStart.ipynb
-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.4K Jul 15 04:28 settings.py
I had tried to install idmlaser
by running pip install -e .
from within the jb/
directory
Harumph. No repro for me on quick test. Will recheck first thing in morning.
Appreciated. All I should need is the jb/
directory, right? Do I need to use the idm pypi in the index-url
for the pip install -e .
? It seemed to be workign withough it.
I guess it make sense that there is no update_ages.cpp in the sandbox/ but why is there a makefile?
Appreciated. All I should need is the
jb/
directory, right? Do I need to use the idm pypi in theindex-url
for thepip install -e .
? It seemed to be workign withough it.
Ya, just pip install -e .
should do it.
I guess it make sense that there is no update_ages.cpp in the sandbox/ but why is there a makefile?
I think it was for if you wanted to change your demographics_settings.py (population, number of nodes, for CCS) and get new input files.
update_ages.cpp should be compiled as part of the install.
The files you need to copy from a sandbox
into your COMPS/idmtools inputs folder should be well captured by the contents of inputs
or inputs_ew
here:
├── inputs
│ ├── demographics_settings.py
│ ├── fits.npy
│ ├── modeled_pop.csv.gz
│ └── settings.py
├── inputs_ew
│ ├── attraction_probabilities.csv
│ ├── cbrs_ew.csv
│ ├── cities.csv
│ ├── demographics_settings.py
│ ├── engwal_modeled.csv.gz
│ ├── fits.npy
│ └── settings.py
├── laser.def
├── laser.id
├── run_laser_with_sif.py
├── sifs
│ └── laser.sif
Looking at my
sandbox/
directory, there was noupdate_ages.cpp
installed:-rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 310 Jul 15 04:28 demographics_settings.py -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.2K Jul 15 04:28 eula_binned.csv -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 106K Jul 15 04:28 eula_pops_20yr_noaging.csv -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 360 Jul 15 04:28 fits.npy -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.7K Jul 15 04:44 makefile -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 6.5M Jul 15 04:28 modeled_pop.csv -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.8M Jul 15 04:28 modeled_pop.csv.gz drwxrwxr-x 2 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 4.0K Jul 15 04:28 __pycache__ -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 458K Jul 15 04:28 QuickStart.ipynb -rw-rw-r-- 1 krosenfeld@internal.idm.ctr krosenfeld@internal.idm.ctr 1.4K Jul 15 04:28 settings.py
You only need data files from the sandbox now. All code is in the installed package.
adding notes to issue thread for continuity:
update_ages.so
. Once I added that step by hand I was able to pip installRelevant code below that appears to to be running?
Line 20 should do the equivalent of what the make command was doing to build the cpp code into a .so during the installation.
Are you install from a .whl (wheel) in your .def?
I'm not installing from a .whl, I'm dong a git clone and pip install: https://github.com/krosenfeld-IDM/202407_laser/blob/16a46de7b4fad5187ef0371e97cb833b99d31dbd/jb_jenkins/Dockerfile#L16
@krosenfeld-IDM Should I see if I can find out why update_ages.cpp wasn't getting compiled during install?
Thanks, I'm still seeing the error.
It would be good to fix if continuing development / use from jb_modulify
. Maybe we can discuss on Wednesday what you, @KevinMcCarthyAtIDM , and @clorton 's thoughts are for next steps?
You can try reproducing it by building a clean codespace and then:
cd jb
pip install -e .
cd ..
mkdir workspace
cd workspace/
python -m idmlaser.utils.build_template_workspace
# building in ./EW
cd EW
python -m idmlaser.measles
produces:
Traceback (most recent call last):
File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/workspaces/laser/jb/src/idmlaser/measles.py", line 17, in <module>
from . import sir_numpy_c as model
File "/workspaces/laser/jb/src/idmlaser/sir_numpy_c.py", line 76, in <module>
update_ages_lib = ctypes.CDLL(shared_lib_path)
File "/usr/local/python/3.10.13/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /workspaces/laser/jb/src/idmlaser/update_ages.so: cannot open shared object file: No such file or directory
Thanks, I'm still seeing the error.
It would be good to fix if continuing development / use from
jb_modulify
. Maybe we can discuss on Wednesday what you, @KevinMcCarthyAtIDM , and @clorton 's thoughts are for next steps?You can try reproducing it by building a clean codespace and then:
cd jb pip install -e . cd .. mkdir workspace cd workspace/ python -m idmlaser.utils.build_template_workspace # building in ./EW cd EW python -m idmlaser.measles
produces:
Traceback (most recent call last): File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/python/3.10.13/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/workspaces/laser/jb/src/idmlaser/measles.py", line 17, in <module> from . import sir_numpy_c as model File "/workspaces/laser/jb/src/idmlaser/sir_numpy_c.py", line 76, in <module> update_ages_lib = ctypes.CDLL(shared_lib_path) File "/usr/local/python/3.10.13/lib/python3.10/ctypes/__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: /workspaces/laser/jb/src/idmlaser/update_ages.so: cannot open shared object file: No such file or directory
Thanks. Able to repro now. Also I get an error when injecting infections (import cases) with a dev install, so that's worth looking at. This whole thing with compiled C code in a module is a bit new and I want to make sure we're catching and fixing issues.
Lots of good suggestions from GPT but dev build isn't invoking custom build commands to create .so yet...
Personally I don't do dev installs. I build a .whl and pip install (from file). I'll keep working on the dev install with Custom Commands for compiling C but the build-and-install-wheel is a completely viable "workaround" if you don't mind the two steps.
If I use the .whl does that mean everytime I make a change o the python code I have to build the .whl and pip install from the file?
I find it really useful to be able to make a change to the code and immediately jump into the e.g. running the scrip or the debugger from vscode.
Ya. Sounds like it would be disruptive to your workflow.
Yeah - I think so. Thanks
Fixed with: https://github.com/InstituteforDiseaseModeling/laser/commit/dd5c258248b0ed4a01c3f2a7f4df53d8e06b52ec
@krosenfeld-IDM you can resolve or close this if no further work is neded.
So for the dev install you can either:
cd jb/
pip install -e .
cd src/idmlaser
make update_ages.so
or
python setup.py develop
on jb_modulify/ #[19bba72] I am following the developer version for installing
idmlaser
from https://github.com/InstituteforDiseaseModeling/laser/wiki/Using-idmlaser-as-a-pip-installable-module. It does not appear to be compilingupdate_ages.cpp
.My commands (from within the
jb/
directory):I build the CCS example in
jb/sandbox
and then get the error: