search

AMICI-dev / AMICI

Advanced Multilanguage Interface to CVODES and IDAS
https://amici.readthedocs.io/
Other
109 stars 31 forks source link

Error with ninja when running with amici version 0.18.1 vs 0.16.0 #2144

Closed ahjanee closed 1 year ago

ahjanee commented 1 year ago

When trying to run python code that previously worked with pypesto and amici 0.16.0 under amici 0.18.1, we encountered the following error 

Traceback (most recent call last):
File "/home/htc/tokunola/Optimization/amici_models/recovery_model_60Hz_no_F_F/setup.py", line 61, in <module>
    setup(
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands()                                                                                      
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/amici/custom_commands.py", line 150, in run
    result = super().run()
  File "/home/htc/tokunola/SCRATCH/conda/envs/pesto/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 93, in run
    raise RuntimeError("Required command 'ninja' not found")
RuntimeError: Required command 'ninja' not found

The package cmake_build_extension was not installed during the older installation. If ninja is required for newer versions of amici, it should be included in the prerequisites (maybe with some further guidance as we could not get it to work even after installation of ninja). For now, we fixed it by simply downgrading to amici 0.16.0.

This was on Debian GNU/Linux 11 (bullseye).

dweindl commented 1 year ago

Hi @ahjanee, those packages are listed as build-system requirements. How are you trying to install amici? pip install amici should take care of all that. If not, this might be related to some ancient pip or setuptools.

ahjanee commented 1 year ago

I did use pip install amici. I checked and I have pip 23.1.2 and setuptools 68.0.0.

dweindl commented 1 year ago

That's weird. Could you please share the complete output of pip install -vvv amici?

ahjanee commented 1 year ago

Using pip 23.1.2 from /home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/ site-packages/pip (python 3.9) Non-user install because site-packages writeable Created temporary directory: /tmp/pip-build-tracker-nbl0ow7w Initialized build tracking at /tmp/pip-build-tracker-nbl0ow7w Created build tracker: /tmp/pip-build-tracker-nbl0ow7w Entered build tracker: /tmp/pip-build-tracker-nbl0ow7w Created temporary directory: /tmp/pip-install-c26e6dej Created temporary directory: /tmp/pip-ephem-wheel-cache-1ki1ur7i Requirement already satisfied: amici in ./miniconda3/envs/Tolu39_1/lib/python3.9 /site-packages (0.16.0) Requirement already satisfied: sympy>=1.9 in ./miniconda3/envs/Tolu39_1/lib/pyth on3.9/site-packages (from amici) (1.12) Requirement already satisfied: python-libsbml in ./miniconda3/envs/Tolu39_1/lib/ python3.9/site-packages (from amici) (5.20.1) Requirement already satisfied: h5py in ./miniconda3/envs/Tolu39_1/lib/python3.9/ site-packages (from amici) (3.9.0) Requirement already satisfied: pandas in ./miniconda3/envs/Tolu39_1/lib/python3. 9/site-packages (from amici) (2.0.3) Requirement already satisfied: pkgconfig in ./miniconda3/envs/Tolu39_1/lib/pytho n3.9/site-packages (from amici) (1.5.5) Requirement already satisfied: wurlitzer in ./miniconda3/envs/Tolu39_1/lib/pytho n3.9/site-packages (from amici) (3.0.3) Requirement already satisfied: toposort in ./miniconda3/envs/Tolu39_1/lib/python 3.9/site-packages (from amici) (1.10) Requirement already satisfied: setuptools>=48 in ./miniconda3/envs/Tolu39_1/lib/ python3.9/site-packages (from amici) (68.0.0) Requirement already satisfied: mpmath in ./miniconda3/envs/Tolu39_1/lib/python3. 9/site-packages (from amici) (1.3.0) Requirement already satisfied: numpy>=1.19.3 in ./miniconda3/envs/Tolu39_1/lib/p ython3.9/site-packages (from amici) (1.25.1) Requirement already satisfied: python-dateutil>=2.8.2 in ./miniconda3/envs/Tolu3 9_1/lib/python3.9/site-packages (from pandas->amici) (2.8.2) Requirement already satisfied: pytz>=2020.1 in ./miniconda3/envs/Tolu39_1/lib/py thon3.9/site-packages (from pandas->amici) (2023.3) Requirement already satisfied: tzdata>=2022.1 in ./miniconda3/envs/Tolu39_1/lib/ python3.9/site-packages (from pandas->amici) (2023.3) Requirement already satisfied: six>=1.5 in ./miniconda3/envs/Tolu39_1/lib/python 3.9/site-packages (from python-dateutil>=2.8.2->pandas->amici) (1.16.0) Created temporary directory: /tmp/pip-unpack-5ij14hz5 Remote version of pip: 23.1.2 Local version of pip: 23.1.2 Was pip installed by pip? False Removed build tracker: '/tmp/pip-build-tracker-nbl0ow7w'

(This was in an env with amici 0.16.0 installed).

dweindl commented 1 year ago

(This was in an env with amici 0.16.0 installed).

Then please run pip uninstall amici before pip install -vvv amici, or try from a new virtual environment.

ahjanee commented 1 year ago

Okay, here is the output after uninstalling amici first:

(Tolu39_1) bzfernsa@htc-login002:~$ pip install -vvv amici Using pip 23.1.2 from /home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/pip (python 3.9) Non-user install because site-packages writeable Created temporary directory: /tmp/pip-build-tracker-wfxp6_xy Initialized build tracking at /tmp/pip-build-tracker-wfxp6_xy Created build tracker: /tmp/pip-build-tracker-wfxp6_xy Entered build tracker: /tmp/pip-build-tracker-wfxp6_xy Created temporary directory: /tmp/pip-install-pjgx37ir Created temporary directory: /tmp/pip-ephem-wheel-cache-5jyqw_vt 1 location(s) to search for versions of amici:

ahjanee commented 1 year ago

When installing in a new venv, I noticed this in the output:

Found link https://files.pythonhosted.org/packages/21/2d/39c6c57032f786f196502 2563eec60623bb3e1409ade6ad834ff703724f3/smmap-5.0.0.tar.gz (from https://pypi.or g/simple/smmap/) (requires-python:>=3.6), version: 5.0.0 Skipping link: not a file: https://pypi.org/simple/smmap/ Given no hashes to check 12 links for project 'smmap': discarding no candidates Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython->cmake-build-extensi on==0.5.1->amici) Created temporary directory: /tmp/pip-unpack-gar7u2mq Looking up "https://files.pythonhosted.org/packages/6d/01/7caa71608bc29952ae09 b0be63a539e50d2484bc37747797a66a60679856/smmap-5.0.0-py3-none-any.whl" in the ca che Current age based on date: 8807 Ignoring unknown cache-control directive: immutable Freshness lifetime from max-age: 365000000 The response is "fresh", returning cached response 365000000 > 8807 Using cached smmap-5.0.0-py3-none-any.whl (24 kB) Added smmap<6,>=3.0.1 from https://files.pythonhosted.org/packages/6d/01/7caa7 1608bc29952ae09b0be63a539e50d2484bc37747797a66a60679856/smmap-5.0.0-py3-none-any .whl (from gitdb<5,>=4.0.1->GitPython->cmake-build-extension==0.5.1->amici) to b uild tracker '/tmp/pip-build-tracker-njg5p77p' Removed smmap<6,>=3.0.1 from https://files.pythonhosted.org/packages/6d/01/7ca a71608bc29952ae09b0be63a539e50d2484bc37747797a66a60679856/smmap-5.0.0-py3-none-a ny.whl (from gitdb<5,>=4.0.1->GitPython->cmake-build-extension==0.5.1->amici) fr om build tracker '/tmp/pip-build-tracker-njg5p77p' Created temporary directory: /tmp/pip-unpack-hhnr1k4u Installing collected packages: toposort, python-libsbml, ninja, mpmath, cmake, w urlitzer, tzdata, typing-extensions, tomli, sympy, smmap, h5py, setuptools-scm, pandas, gitdb, GitPython, cmake-build-extension, amici

changing mode of /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja to 755

changing mode of /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/cmake to 755 changing mode of /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/cpack to 755 changing mode of /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ctest to 755

changing mode of /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/isympy to 755

dweindl commented 1 year ago

Installing collected packages: [...] ninja, [...] cmake-build-extension, [...]

Looks good. So everything works now?

ahjanee commented 1 year ago

No, when I run now this gives the same error as in my initial report. :/

dweindl commented 1 year ago

Okay, I am starting to get it. So the problem is not installing AMICI, but importing a model, right?

Can you please show the code that caused the initial error? (removing any sensitive information; no actual model files are needed)

File "/home/htc/tokunola/Optimization/amici_models/recovery_model_60Hz_no_F_F/setup.py", line 61, in <module>
    setup(

These line numbers seem to indicate that this model was generated by an older AMICI version. The generated model code is usually not compatible with any other AMICI version. Consider deleting /home/htc/tokunola/Optimization/amici_models/recovery_model_60Hz_no_F_F and re-importing the model.

ahjanee commented 1 year ago

Deleting the amici_models directory and running again produces the same error. This is probably the relevant part from my code:


# Import model

petab_problem = petab.Problem.from_yaml(model_path)

importer = pypesto.petab.PetabImporter(petab_problem)

model = importer.create_model(
    simplify=False, 
    generate_sensitivity_code=True,
)

objective = importer.create_objective()

Yes, I think the error happens during model import/building, this is the log:

> Compiling amici model to folder /home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F.
> 2023-07-11 13:46:39.038 - amici.petab_import - INFO - Importing model ...
> 2023-07-11 13:46:39.038 - amici.petab_import - INFO - Validating PEtab problem ...
> Visualization table not available. Skipping.
> 2023-07-11 13:46:39.664 - amici.petab_import - INFO - Model name is 'recovery_model_60Hz_no_F_F'.
> Writing model code to '/home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F'.
> 2023-07-11 13:46:39.669 - amici.petab_import - INFO - Species: 7
> 2023-07-11 13:46:39.669 - amici.petab_import - INFO - Global parameters: 217
> 2023-07-11 13:46:39.669 - amici.petab_import - INFO - Reactions: 0
> 2023-07-11 13:47:00.332 - amici.petab_import - INFO - Observables: 1
> 2023-07-11 13:47:00.332 - amici.petab_import - INFO - Sigmas: 1
> 2023-07-11 13:47:00.353 - amici.petab_import - DEBUG - Adding output parameters to model: ['noiseParameter1_rel']
> 2023-07-11 13:47:00.354 - amici.petab_import - DEBUG - Adding initial assignments for dict_keys([])
> 2023-07-11 13:47:00.490 - amici.petab_import - DEBUG - Condition table: (1, 1)
> 2023-07-11 13:47:00.490 - amici.petab_import - WARNING - Column 'noiseParameter1_rel' used in condition table but not entity with the corresponding ID exists. Ignoring.
> 2023-07-11 13:47:00.491 - amici.petab_import - DEBUG - Fixed parameters are ['L', 'a1', 'a10', 'a100', 'a11', 'a12', 'a13', 'a14', 'a15', 'a16', 'a17', 'a18', 'a19', 'a2', 'a20', 'a21', 'a22', 'a23', 'a24', 'a25', 'a26', 'a27', 'a28', 'a29', 'a3', 'a30', 'a31', 'a32', 'a33', 'a34', 'a35', 'a36', 'a37', 'a38', 'a39', 'a4', 'a40', 'a41', 'a42', 'a43', 'a44', 'a45', 'a46', 'a47', 'a48', 'a49', 'a5', 'a50', 'a51', 'a52', 'a53', 'a54', 'a55', 'a56', 'a57', 'a58', 'a59', 'a6', 'a60', 'a61', 'a62', 'a63', 'a64', 'a65', 'a66', 'a67', 'a68', 'a69', 'a7', 'a70', 'a71', 'a72', 'a73', 'a74', 'a75', 'a76', 'a77', 'a78', 'a79', 'a8', 'a80', 'a81', 'a82', 'a83', 'a84', 'a85', 'a86', 'a87', 'a88', 'a89', 'a9', 'a90', 'a91', 'a92', 'a93', 'a94', 'a95', 'a96', 'a97', 'a98', 'a99', 'cliffstart', 'fac', 'kUmax', 'kUmin', 'k_base', 'mu1', 'mu10', 'mu100', 'mu11', 'mu12', 'mu13', 'mu14', 'mu15', 'mu16', 'mu17', 'mu18', 'mu19', 'mu2', 'mu20', 'mu21', 'mu22', 'mu23', 'mu24', 'mu25', 'mu26', 'mu27', 'mu28', 'mu29', 'mu3', 'mu30', 'mu31', 'mu32', 'mu33', 'mu34', 'mu35', 'mu36', 'mu37', 'mu38', 'mu39', 'mu4', 'mu40', 'mu41', 'mu42', 'mu43', 'mu44', 'mu45', 'mu46', 'mu47', 'mu48', 'mu49', 'mu5', 'mu50', 'mu51', 'mu52', 'mu53', 'mu54', 'mu55', 'mu56', 'mu57', 'mu58', 'mu59', 'mu6', 'mu60', 'mu61', 'mu62', 'mu63', 'mu64', 'mu65', 'mu66', 'mu67', 'mu68', 'mu69', 'mu7', 'mu70', 'mu71', 'mu72', 'mu73', 'mu74', 'mu75', 'mu76', 'mu77', 'mu78', 'mu79', 'mu8', 'mu80', 'mu81', 'mu82', 'mu83', 'mu84', 'mu85', 'mu86', 'mu87', 'mu88', 'mu89', 'mu9', 'mu90', 'mu91', 'mu92', 'mu93', 'mu94', 'mu95', 'mu96', 'mu97', 'mu98', 'mu99', 'nsites', 'nves', 'sigma', 'steep', 't0', 't_wait']
> 2023-07-11 13:47:00.491 - amici.petab_import - INFO - Overall fixed parameters: 212
> 2023-07-11 13:47:00.491 - amici.petab_import - INFO - Variable parameters: 6
> 2023-07-11 13:47:00.492 - amici.sbml_import - DEBUG - Finished processing SBML annotations        ++ (4.40E-04s)
> 2023-07-11 13:47:00.496 - amici.sbml_import - DEBUG - Finished gathering local SBML symbols       ++ (3.95E-03s)
> 2023-07-11 13:47:00.501 - amici.sbml_import - DEBUG - Finished processing SBML parameters         ++ (3.99E-03s)
> 2023-07-11 13:47:00.501 - amici.sbml_import - DEBUG - Finished processing SBML compartments       ++ (1.60E-04s)
> 2023-07-11 13:47:03.049 - amici.sbml_import - DEBUG - Finished processing SBML species initials  +++ (2.55E+00s)
> 2023-07-11 13:47:06.502 - amici.sbml_import - DEBUG - Finished processing SBML rate rules        +++ (3.45E+00s)
> 2023-07-11 13:47:06.502 - amici.sbml_import - DEBUG - Finished processing SBML species            ++ (6.00E+00s)
> 2023-07-11 13:47:06.504 - amici.sbml_import - DEBUG - Finished processing SBML reactions          ++ (1.73E-04s)
> 2023-07-11 13:47:06.537 - amici.sbml_import - DEBUG - Finished processing SBML rules              ++ (3.34E-02s)
> 2023-07-11 13:47:06.538 - amici.sbml_import - DEBUG - Finished processing SBML initial assignments++ (1.55E-04s)
> 2023-07-11 13:47:06.538 - amici.sbml_import - DEBUG - Finished processing SBML species references ++ (9.75E-06s)
> 2023-07-11 13:47:06.538 - amici.sbml_import - DEBUG - Finished processing SBML events             ++ (4.21E-05s)
> 2023-07-11 13:47:06.538 - amici.sbml_import - DEBUG - Finished importing SBML                      + (6.05E+00s)
> 2023-07-11 13:47:06.729 - amici.sbml_import - DEBUG - Finished processing SBML observables         + (1.91E-01s)
> 2023-07-11 13:47:06.729 - amici.sbml_import - DEBUG - Finished processing SBML event observables   + (1.31E-06s)
> 2023-07-11 13:47:07.333 - amici.de_export - DEBUG - Finished running smart_multiply               ++ (4.06E-05s)
> 2023-07-11 13:47:15.035 - amici.de_export - DEBUG - Finished importing SbmlImporter                + (7.70E+00s)
> 2023-07-11 13:47:15.038 - amici.de_export - DEBUG - Finished computing Jy                        +++ (3.19E-04s)
> 2023-07-11 13:47:15.039 - amici.de_export - DEBUG - Finished computing y                         +++ (6.13E-05s)
> 2023-07-11 13:47:15.397 - amici.de_export - DEBUG - Finished computing sigmay                    +++ (1.12E-04s)
> 2023-07-11 13:47:15.407 - amici.de_export - DEBUG - Finished writing Jy.cpp                       ++ (3.70E-01s)
> 2023-07-11 13:47:15.416 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (7.71E-03s)
> 2023-07-11 13:47:15.416 - amici.de_export - DEBUG - Finished computing dJydsigma                 +++ (8.28E-03s)
> 2023-07-11 13:47:15.418 - amici.de_export - DEBUG - Finished writing dJydsigma.cpp                ++ (1.01E-02s)
> 2023-07-11 13:47:15.422 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (3.25E-03s)
> 2023-07-11 13:47:15.422 - amici.de_export - DEBUG - Finished computing dJydy                     +++ (3.51E-03s)
> 2023-07-11 13:47:15.423 - amici.de_export - DEBUG - Finished writing dJydy.cpp                    ++ (5.32E-03s)
> 2023-07-11 13:47:15.425 - amici.de_export - DEBUG - Finished computing Jz                        +++ (7.57E-05s)
> 2023-07-11 13:47:15.425 - amici.de_export - DEBUG - Finished computing z                         +++ (3.76E-05s)
> 2023-07-11 13:47:15.425 - amici.de_export - DEBUG - Finished computing sigmaz                    +++ (4.85E-05s)
> 2023-07-11 13:47:15.425 - amici.de_export - DEBUG - Finished writing Jz.cpp                       ++ (5.88E-04s)
> 2023-07-11 13:47:15.425 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (2.18E-05s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished computing dJzdsigma                 +++ (2.05E-04s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished writing dJzdsigma.cpp                ++ (3.45E-04s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.92E-05s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished computing dJzdz                     +++ (1.91E-04s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished writing dJzdz.cpp                    ++ (3.26E-04s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished computing Jrz                       +++ (4.81E-05s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished computing rz                        +++ (3.04E-05s)
> 2023-07-11 13:47:15.426 - amici.de_export - DEBUG - Finished writing Jrz.cpp                      ++ (3.36E-04s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.87E-05s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished computing dJrzdsigma                +++ (2.04E-04s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished writing dJrzdsigma.cpp               ++ (3.41E-04s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.91E-05s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished computing dJrzdz                    +++ (1.94E-04s)
> 2023-07-11 13:47:15.427 - amici.de_export - DEBUG - Finished writing dJrzdz.cpp                   ++ (3.28E-04s)
> 2023-07-11 13:47:15.428 - amici.de_export - DEBUG - Finished computing root                      +++ (5.47E-05s)
> 2023-07-11 13:47:15.428 - amici.de_export - DEBUG - Finished writing root.cpp                     ++ (1.84E-04s)
> 2023-07-11 13:47:15.428 - amici.de_export - DEBUG - Finished computing w                        ++++ (6.01E-05s)
> 2023-07-11 13:47:15.428 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.11E-04s)
> 2023-07-11 13:47:15.428 - amici.de_export - DEBUG - Finished computing dwdp                      +++ (5.21E-04s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished computing spl                       +++ (3.05E-05s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished computing sspl                      +++ (2.73E-05s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished writing dwdp.cpp                     ++ (1.07E-03s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (8.78E-05s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished computing dwdx                      +++ (3.75E-04s)
> 2023-07-11 13:47:15.429 - amici.de_export - DEBUG - Finished writing dwdx.cpp                     ++ (6.10E-04s)
> 2023-07-11 13:47:15.430 - amici.de_export - DEBUG - Finished writing create_splines.cpp           ++ (5.39E-04s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished computing spline_values            ++++ (4.50E-05s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.95E-05s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished computing dspline_valuesdp          +++ (3.95E-04s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished writing dspline_valuesdp.cpp         ++ (5.20E-04s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished computing spline_slopes            ++++ (4.24E-05s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.83E-05s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished computing dspline_slopesdp          +++ (3.79E-04s)
> 2023-07-11 13:47:15.431 - amici.de_export - DEBUG - Finished writing dspline_slopesdp.cpp         ++ (5.03E-04s)
> 2023-07-11 13:47:15.432 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (7.71E-05s)
> 2023-07-11 13:47:15.432 - amici.de_export - DEBUG - Finished computing dwdw                      +++ (4.51E-04s)
> 2023-07-11 13:47:15.432 - amici.de_export - DEBUG - Finished writing dwdw.cpp                     ++ (6.61E-04s)
> 2023-07-11 13:47:15.433 - amici.de_export - DEBUG - Finished computing xdot                     ++++ (6.79E-05s)
> 2023-07-11 13:47:16.087 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (6.55E-01s)
> 2023-07-11 13:47:16.088 - amici.de_export - DEBUG - Finished computing dxdotdw                   +++ (6.55E-01s)
> 2023-07-11 13:47:16.587 - amici.de_export - DEBUG - Finished writing dxdotdw.cpp                  ++ (1.15E+00s)
> 2023-07-11 13:47:16.718 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (1.29E-01s)
> 2023-07-11 13:47:16.718 - amici.de_export - DEBUG - Finished computing dxdotdx_explicit          +++ (1.29E-01s)
> 2023-07-11 13:47:16.964 - amici.de_export - DEBUG - Finished writing dxdotdx_explicit.cpp         ++ (3.76E-01s)
> 2023-07-11 13:47:17.001 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (3.55E-02s)
> 2023-07-11 13:47:17.001 - amici.de_export - DEBUG - Finished computing dxdotdp_explicit          +++ (3.58E-02s)
> 2023-07-11 13:47:17.050 - amici.de_export - DEBUG - Finished writing dxdotdp_explicit.cpp         ++ (8.47E-02s)
> 2023-07-11 13:47:17.055 - amici.de_export - DEBUG - Finished running smart_jacobian            +++++ (3.27E-03s)
> 2023-07-11 13:47:17.055 - amici.de_export - DEBUG - Finished computing dydx                     ++++ (3.50E-03s)
> 2023-07-11 13:47:17.059 - amici.de_export - DEBUG - Finished running smart_jacobian            +++++ (3.82E-03s)
> 2023-07-11 13:47:17.060 - amici.de_export - DEBUG - Finished running smart_multiply            +++++ (2.35E-05s)
> 2023-07-11 13:47:17.060 - amici.de_export - DEBUG - Finished computing dydw                     ++++ (4.29E-03s)
> 2023-07-11 13:47:17.060 - amici.de_export - DEBUG - Finished computing dydx                      +++ (8.67E-03s)
> 2023-07-11 13:47:17.108 - amici.de_export - DEBUG - Finished writing dydx.cpp                     ++ (5.65E-02s)
> 2023-07-11 13:47:17.112 - amici.de_export - DEBUG - Finished running smart_jacobian            +++++ (3.14E-03s)
> 2023-07-11 13:47:17.112 - amici.de_export - DEBUG - Finished computing dydp                     ++++ (3.37E-03s)
> 2023-07-11 13:47:17.112 - amici.de_export - DEBUG - Finished computing dydp                      +++ (3.89E-03s)
> 2023-07-11 13:47:17.160 - amici.de_export - DEBUG - Finished writing dydp.cpp                     ++ (5.21E-02s)
> 2023-07-11 13:47:17.160 - amici.de_export - DEBUG - Finished computing dzdx                      +++ (5.50E-05s)
> 2023-07-11 13:47:17.160 - amici.de_export - DEBUG - Finished writing dzdx.cpp                     ++ (2.01E-04s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished computing dzdp                      +++ (3.96E-05s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished writing dzdp.cpp                     ++ (1.70E-04s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished computing drzdx                     +++ (3.48E-05s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished writing drzdx.cpp                    ++ (1.62E-04s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished computing drzdp                     +++ (3.25E-05s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished writing drzdp.cpp                    ++ (1.60E-04s)
> 2023-07-11 13:47:17.161 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (6.51E-05s)
> 2023-07-11 13:47:17.162 - amici.de_export - DEBUG - Finished computing dsigmaydy                 +++ (2.42E-04s)
> 2023-07-11 13:47:17.162 - amici.de_export - DEBUG - Finished writing dsigmaydy.cpp                ++ (4.00E-04s)
> 2023-07-11 13:47:17.162 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (2.00E-04s)
> 2023-07-11 13:47:17.162 - amici.de_export - DEBUG - Finished computing dsigmaydp                 +++ (3.90E-04s)
> 2023-07-11 13:47:17.163 - amici.de_export - DEBUG - Finished writing dsigmaydp.cpp                ++ (1.50E-03s)
> 2023-07-11 13:47:17.164 - amici.de_export - DEBUG - Finished writing sigmay.cpp                   ++ (7.99E-04s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (2.23E-05s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished computing dsigmazdp                 +++ (2.16E-04s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished writing dsigmazdp.cpp                ++ (3.47E-04s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished writing sigmaz.cpp                   ++ (1.16E-05s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished computing stau                      +++ (3.88E-05s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished writing stau.cpp                     ++ (1.72E-04s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished computing deltax                    +++ (3.30E-05s)
> 2023-07-11 13:47:17.165 - amici.de_export - DEBUG - Finished writing deltax.cpp                   ++ (1.81E-04s)
> 2023-07-11 13:47:17.166 - amici.de_export - DEBUG - Finished computing deltasx                   +++ (3.46E-05s)
> 2023-07-11 13:47:17.166 - amici.de_export - DEBUG - Finished writing deltasx.cpp                  ++ (1.91E-04s)
> 2023-07-11 13:47:17.167 - amici.de_export - DEBUG - Finished writing w.cpp                        ++ (9.70E-04s)
> 2023-07-11 13:47:17.167 - amici.de_export - DEBUG - Finished computing x0                        +++ (1.01E-04s)
> 2023-07-11 13:47:19.979 - amici.de_export - DEBUG - Finished writing x0.cpp                       ++ (2.81E+00s)
> 2023-07-11 13:47:20.037 - amici.de_export - DEBUG - Finished computing x0_fixedParameters        +++ (5.70E-02s)
> 2023-07-11 13:47:22.669 - amici.de_export - DEBUG - Finished writing x0_fixedParameters.cpp       ++ (2.69E+00s)
> 2023-07-11 13:47:25.196 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (2.53E+00s)
> 2023-07-11 13:47:25.197 - amici.de_export - DEBUG - Finished computing sx0                       +++ (2.53E+00s)
> 2023-07-11 13:47:44.132 - amici.de_export - DEBUG - Finished writing sx0.cpp                      ++ (2.15E+01s)
> 2023-07-11 13:47:44.845 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (7.11E-01s)
> 2023-07-11 13:47:44.845 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (2.18E-04s)
> 2023-07-11 13:47:44.845 - amici.de_export - DEBUG - Finished computing sx0_fixedParameters       +++ (7.12E-01s)
> 2023-07-11 13:48:04.960 - amici.de_export - DEBUG - Finished writing sx0_fixedParameters.cpp      ++ (2.08E+01s)
> 2023-07-11 13:48:05.267 - amici.de_export - DEBUG - Finished writing xdot.cpp                     ++ (3.06E-01s)
> 2023-07-11 13:48:05.326 - amici.de_export - DEBUG - Finished writing y.cpp                        ++ (5.84E-02s)
> 2023-07-11 13:48:05.326 - amici.de_export - DEBUG - Finished computing x_rdata                   +++ (2.81E-04s)
> 2023-07-11 13:48:05.328 - amici.de_export - DEBUG - Finished writing x_rdata.cpp                  ++ (1.70E-03s)
> 2023-07-11 13:48:05.328 - amici.de_export - DEBUG - Finished computing total_cl                  +++ (1.40E-04s)
> 2023-07-11 13:48:05.328 - amici.de_export - DEBUG - Finished writing total_cl.cpp                 ++ (3.59E-04s)
> 2023-07-11 13:48:05.329 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (9.05E-05s)
> 2023-07-11 13:48:05.329 - amici.de_export - DEBUG - Finished computing dtotal_cldp               +++ (3.38E-04s)
> 2023-07-11 13:48:05.329 - amici.de_export - DEBUG - Finished writing dtotal_cldp.cpp              ++ (5.14E-04s)
> 2023-07-11 13:48:05.329 - amici.de_export - DEBUG - Finished computing dtotal_cldx_rdata         +++ (6.84E-05s)
> 2023-07-11 13:48:05.329 - amici.de_export - DEBUG - Finished writing dtotal_cldx_rdata.cpp        ++ (3.68E-04s)
> 2023-07-11 13:48:05.330 - amici.de_export - DEBUG - Finished computing x_solver                  +++ (8.40E-05s)
> 2023-07-11 13:48:05.331 - amici.de_export - DEBUG - Finished writing x_solver.cpp                 ++ (1.46E-03s)
> 2023-07-11 13:48:05.331 - amici.de_export - DEBUG - Finished computing dx_rdatadx_solver         +++ (1.10E-04s)
> 2023-07-11 13:48:05.332 - amici.de_export - DEBUG - Finished writing dx_rdatadx_solver.cpp        ++ (7.06E-04s)
> 2023-07-11 13:48:05.332 - amici.de_export - DEBUG - Finished computing dx_rdatadp                +++ (5.72E-05s)
> 2023-07-11 13:48:05.332 - amici.de_export - DEBUG - Finished writing dx_rdatadp.cpp               ++ (3.23E-04s)
> 2023-07-11 13:48:05.333 - amici.de_export - DEBUG - Finished running smart_jacobian             ++++ (3.62E-05s)
> 2023-07-11 13:48:05.333 - amici.de_export - DEBUG - Finished computing dx_rdatadtcl              +++ (2.27E-04s)
> 2023-07-11 13:48:05.333 - amici.de_export - DEBUG - Finished writing dx_rdatadtcl.cpp             ++ (4.26E-04s)
> 2023-07-11 13:48:05.333 - amici.de_export - DEBUG - Finished writing z.cpp                        ++ (1.60E-05s)
> 2023-07-11 13:48:05.333 - amici.de_export - DEBUG - Finished writing rz.cpp                       ++ (1.47E-05s)
> 2023-07-11 13:48:05.384 - amici.de_export - DEBUG - Finished generating cpp code                   + (5.03E+01s)
> running AmiciInstall
> running build_ext
> running AmiciBuildCMakeExtension
> Traceback (most recent call last):
>   File "/home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F/setup.py", line 61, in <module>
>     setup(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
>     return distutils.core.setup(**attrs)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
>     return run_commands(dist)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
>     dist.run_commands()
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
>     self.run_command(cmd)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
>     super().run_command(command)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
>     cmd_obj.run()
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/custom_commands.py", line 150, in run
>     result = super().run()
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 93, in run
>     raise RuntimeError("Required command 'ninja' not found")
> RuntimeError: Required command 'ninja' not found
> 
> Failed building the model extension.
> Traceback (most recent call last):
>   File "/home/htc/bzfernsa/PMU4_1_Tolu1/optimization.py", line 47, in <module>
>     model = importer.create_model(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/pypesto/petab/importer.py", line 279, in create_model
>     self.compile_model(**kwargs)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/pypesto/petab/importer.py", line 346, in compile_model
>     amici.petab_import.import_petab_problem(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/petab_import.py", line 356, in import_petab_problem
>     import_model_sbml(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/logging.py", line 209, in wrapper_timer
>     rval = func(*args, **kwargs)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/petab_import.py", line 746, in import_model_sbml
>     sbml_importer.sbml2amici(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/sbml_import.py", line 424, in sbml2amici
>     exporter.compile_model()
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/logging.py", line 209, in wrapper_timer
>     rval = func(*args, **kwargs)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/de_export.py", line 2724, in compile_model
>     self._compile_c_code(compiler=self.compiler, verbose=self.verbose)
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/site-packages/amici/de_export.py", line 2820, in _compile_c_code
>     result = subprocess.run(
>   File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/lib/python3.9/subprocess.py", line 528, in run
>     raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['/home/htc/bzfernsa/miniconda3/envs/Tolu39_1/bin/python', '/home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F/setup.py', '--verbose', 'build_ext', '--build-lib=/home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F', '--build-temp=/home/htc/bzfernsa/PMU4_1_Tolu1/amici_models/recovery_model_60Hz_no_F_F/build']' returned non-zero exit status 1.
> --------------------------------------------------------------------------
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> mpiexec detected that one or more processes exited with non-zero status, thus causing
> the job to be terminated. The first process to do so was:
> 
>   Process name: [[21522,1],0]
>   Exit code:    1
> --------------------------------------------------------------------------
dweindl commented 1 year ago

Okay, this is strange. Ninja is installed when AMICI get installed, but later it is not found anymore.

What's the output of pip show ninja && which ninja && ninja --version on the shell, in your respective python environment?

ahjanee commented 1 year ago

The output is

(Tolu39_2) bzfernsa@htc-login002:~/PMU4_1_Tolu1$ pip show ninja && which ninja && ninja --version
Name: ninja
Version: 1.11.1
Summary: Ninja is a small build system with a focus on speed
Home-page: http://ninja-build.org/
Author: Jean-Christophe Fillion-Robin
Author-email: scikit-build@googlegroups.com
License: Apache 2.0
Location: /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/lib/python3.9/site-packages
Requires:
Required-by: cmake-build-extension
/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja
1.11.1.git.kitware.jobserver-1
dweindl commented 1 year ago

Looks good. python -c 'import shutil; print(shutil.which("ninja"))'?

ahjanee commented 1 year ago 
(Tolu39_2) bzfernsa@htc-login002:~/PMU4_1_Tolu1$ python -c 'import shutil; print(shutil.which("ninja"))'
/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja
dweindl commented 1 year ago 
(Tolu39_2) bzfernsa@htc-login002:~/PMU4_1_Tolu1$ python -c 'import shutil; print(shutil.which("ninja"))'
/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja

Generally good. Strange though, because that's precisely what fails in your initial post:

https://github.com/diegoferigo/cmake-build-extension/blob/2c3d822e40fe3fd2d769ad09dba367e6617dcc82/src/cmake_build_extension/build_extension.py#L91-L93

I assume the switch from environment Tolu39_1 to Tolu39_2 was intentional?!

So in a fresh python environment the following code still gives the initial error? 

pip install amici[petab]
amici_import_petab -v -y /path/to/some/petab.yaml  # <-- adapt
ahjanee commented 1 year ago

I assume the switch from environment Tolu39_1 to Tolu39_2 was intentional?!

Yes, that was the new venv I set up for the installation before.

So in a fresh python environment the following code still gives the initial error?


pip install amici[petab]
amici_import_petab -v -y /path/to/some/petab.yaml  # <-- adapt

This did not produce an error for me. But I get the error in a fresh environment (+amici and pypesto) for my code.

dweindl commented 1 year ago

This did not produce an error for me. But I get the error in a fresh environment (+amici and pypesto) for my code.

This sounds like a difference in environment variables when you are running the one versus the other.

Is there a difference between the output of echo $PATH on the shell and import os; print(os.getenv('PATH')) from the python session from which pypesto failed?

dweindl commented 1 year ago

Am I interpreting this correctly?

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

This happened on some allocated compute node on a cluster?

(Tolu39_2) bzfernsa@htc-login002:~/PMU4_1_Tolu1$ python -c 'import shutil; print(shutil.which("ninja"))' /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja

This was during an interactive session on a login node?

Depending on the scheduler, $PATH from your submission environment may or may not be exported when your job gets launched. There might be options to change that, or you might have to add /home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ to your $PATH yourself.

ahjanee commented 1 year ago

This sounds like a difference in environment variables when you are running the one versus the other.

I think that might be it. Yes, the problem arises during scheduled job execution on a cluster. I tried fixing the problem by appending to $PATH from inside my python program via os.environ["PATH"] and I also set some other environment variables that way (os.environ["LSWIG"], os.environ["LD_LIBRARY_PATH"]). Now, the error log changed:

2023-07-14 14:02:12.778 - amici.de_export - DEBUG - Finished generating cpp code + (9.22E+01s) running AmiciInstall running build_ext ------------------------------ model_ext ------------------------------ -- The C compiler identification is GNU 10.2.1 -- The CXX compiler identification is GNU 10.2.1 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Performing Test -Wall -- Performing Test -Wall - Failed -- Performing Test -Wno-unused-function -- Performing Test -Wno-unused-function - Failed -- Performing Test -Wno-unused-variable -- Performing Test -Wno-unused-variable - Failed -- Performing Test -Wno-unused-but-set-variable -- Performing Test -Wno-unused-but-set-variable - Failed -- Found OpenMP_C: -fopenmp (found version "4.5") -- Found OpenMP_CXX: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5")
-- original library suffixes: .so;.a -- revised for static search: .a;.so;.a -- major: #define SUITESPARSE_MAIN_VERSION 7 -- minor: #define SUITESPARSE_SUB_VERSION 0 -- patch: #define SUITESPARSE_SUBSUB_VERSION 1 -- Found SuiteSparse_config: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libsuitesparseconfig.so (found version "7.0.1") -- result: -- SuiteSparse_config version: 7.0.1 -- SuiteSparse_config include: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/include -- SuiteSparse_config library: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libsuitesparseconfig.so -- SuiteSparse_config static: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libsuitesparseconfig.a -- major: #define AMD_MAIN_VERSION 3 -- minor: #define AMD_SUB_VERSION 0 -- patch: #define AMD_SUBSUB_VERSION 3 -- Found AMD: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libamd.so (found version "3.0.3") -- AMD version: 3.0.3 -- AMD include: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/include -- AMD library: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libamd.so -- AMD static: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libamd.a -- major: #define BTF_MAIN_VERSION 2 -- minor: #define BTF_SUB_VERSION 0 -- patch: #define BTF_SUBSUB_VERSION 3 -- Found BTF: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libbtf.so (found version "2.0.3") -- BTF version: 2.0.3 -- BTF include: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/include -- BTF library: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libbtf.so -- BTF static: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libbtf.a -- major: #define COLAMD_MAIN_VERSION 3 -- minor: #define COLAMD_SUB_VERSION 0 -- patch: #define COLAMD_SUBSUB_VERSION 3 -- Found COLAMD: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libcolamd.so (found version "3.0.3") -- COLAMD version: 3.0.3 -- COLAMD include: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/include -- COLAMD library: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libcolamd.so -- COLAMD static: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libcolamd.a -- major: #define KLU_MAIN_VERSION 2 -- minor: #define KLU_SUB_VERSION 0 -- patch: #define KLU_SUBSUB_VERSION 3 -- Found KLU: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libklu.so (found version "2.0.3") -- KLU version: 2.0.3 -- KLU include: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/include -- KLU library: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libklu.so -- KLU static: /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/libklu.a -- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0") found components: chrono -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5_cpp.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/home/htc/bzfernsa/miniconda3/lib/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.6") found components: C HL CXX -- Found AMICI /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/cmake/Amici -- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0")
-- Found SWIG: /home/htc/bzfernsa/swig-4.1.1/swig (found version "4.1.1")
-- Found Python3: /usr/bin/python3.9 (found version "3.9.2") found components: Interpreter Development Development.Module Development.Embed -- Python extension suffix is .cpython-39-x86_64-linux-gnu.so -- Configuring done CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library): Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): swig/CMakeLists.txt:26 (swig_add_library)

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:86 (add_library): Target "recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

-- Generating done CMake Generate step failed. Build files cannot be regenerated correctly. running AmiciBuildCMakeExtension

==> Configuring: $ cmake -S /home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F -B /home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/build_model_ext -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/recovery_model_60Hz_no_F_F -DCMAKE_MAKE_PROGRAM=/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_MODULE_PATH=/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/cmake/SuiteSparse;/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib64/cmake/SuiteSparse -DKLU_ROOT=/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici -DAMICI_PYTHON_BUILD_EXT_ONLY=ON

==> Building: $ cmake --build /home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/build_model_ext --config Release

==> Installing: $ cmake --install /home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/build_model_ext

Traceback (most recent call last): File "/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/setup.py", line 61, in setup( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/init.py", line 107, in setup return distutils.core.setup(**attrs) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command super().run_command(command) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/custom_commands.py", line 150, in run result = super().run() File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 113, in run self.build_extension(ext) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/custom_commands.py", line 177, in build_extension super().build_extension(ext) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/cmake_build_extension/build_extension.py", line 229, in build_extension subprocess.check_call(configure_command) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '-S', '/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F', '-B', '/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/build_model_ext', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/recovery_model_60Hz_no_F_F', '-DCMAKE_MAKE_PROGRAM=/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/ninja', '-DCMAKE_VERBOSE_MAKEFILE=ON', '-DCMAKE_MODULE_PATH=/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/cmake/SuiteSparse;/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib64/cmake/SuiteSparse', '-DKLU_ROOT=/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici', '-DAMICI_PYTHON_BUILD_EXT_ONLY=ON']' returned non-zero exit status 1.

Failed building the model extension. Traceback (most recent call last): File "/home/htc/bzfernsa/PMU4_1_Tolu2/test.py", line 55, in model = importer.create_model( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/pypesto/petab/importer.py", line 279, in create_model self.compile_model(kwargs) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/pypesto/petab/importer.py", line 346, in compile_model amici.petab_import.import_petab_problem( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/petab_import.py", line 356, in import_petab_problem import_model_sbml( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/logging.py", line 209, in wrapper_timer rval = func(*args, *kwargs) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/petab_import.py", line 746, in import_model_sbml sbml_importer.sbml2amici( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/sbml_import.py", line 424, in sbml2amici exporter.compile_model() File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/logging.py", line 209, in wrapper_timer rval = func(args, kwargs) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/de_export.py", line 2724, in compile_model self._compile_c_code(compiler=self.compiler, verbose=self.verbose) File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/de_export.py", line 2820, in _compile_c_code result = subprocess.run( File "/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/bin/python', '/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/setup.py', '--verbose', 'build_ext', '--build-lib=/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F', '--build-temp=/home/htc/bzfernsa/PMU4_1_Tolu2/amici_models/recovery_model_60Hz_no_F_F/build']' returned non-zero exit status 1.

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

mpiexec detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:

Process name: [[255,1],0] Exit code: 1

For comparison, the output on the shell in response to echo $PATH is this

/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/bin:/home/htc/bzfernsa/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/condabin:/home/htc/bzfernsa/miniconda3/lib:/home/htc/bzfernsa/miniconda3:/home/htc/bzfernsa/julia-1.6.1/bin:/home/htc/bzfernsa/swig-4.1.1:/home/htc/bzfernsa/swig/bin

while the import os; print(os.getenv('PATH')) inside the job now gives

/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/envs/test/bin:/home/htc/bzfernsa/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/condabin:/home/htc/bzfernsa/miniconda3/lib:/home/htc/bzfernsa/miniconda3:/home/htc/bzfernsa/julia-1.6.1/bin:/home/htc/bzfernsa/swig-4.1.1:/home/htc/bzfernsa/swig/bin:/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/

dweindl commented 1 year ago

There still seems to be some PATH / LD_LIBRARY_DIR issues left:

CMake Error at /usr/share/cmake-3.18/Modules/UseSWIG.cmake:750 (add_library):

This indicates that the cmake installed via pip is not used, but your system cmake. This is not necessarily an issue in itself, but might cause problems elsewhere. Either the virtual environment bin/ is not included in PATH, or in the wrong order.

Target "_recovery_model_60Hz_no_F_F" links to target "hdf5::hdf5_hl_cpp" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

This might happen if you install the amici package on the login node, where hdf5 is available, but then import a model where hdf5 is not available. Installing amici from the same environment from which you import the model might help. We might add an easier way to enable/disable that.

Can you share your /home/htc/bzfernsa/miniconda3/envs/Tolu39_4/lib/python3.9/site-packages/amici/lib/cmake/Amici/AmiciConfig.cmake?

--

I think the problem here is not so much about amici, but rather about environment management. I am a bit surprised that PATH gets messed up. Are you properly activating your virtual python environment? This should set PATH in a way that the first issue above gets resolved and shouldn't require any manual intervention. Regarding the hdf5 issue, depending on your cluster, this might be resolved by module load hdf5 or similar before importing the model (alternatively module unload hdf5 before installing the amici package). Your cluster documentation might tell you more.

dweindl commented 1 year ago

For comparison, the output on the shell in response to echo $PATH is this

/home/htc/bzfernsa/miniconda3/envs/Tolu39_4/bin:/home/htc/bzfernsa/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/condabin:/home/htc/bzfernsa/miniconda3/lib:/home/htc/bzfernsa/miniconda3:/home/htc/bzfernsa/julia-1.6.1/bin:/home/htc/bzfernsa/swig-4.1.1:/home/htc/bzfernsa/swig/bin

while the import os; print(os.getenv('PATH')) inside the job now gives

/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/envs/test/bin:/home/htc/bzfernsa/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/htc/bzfernsa/miniconda3/condabin:/home/htc/bzfernsa/miniconda3/lib:/home/htc/bzfernsa/miniconda3:/home/htc/bzfernsa/julia-1.6.1/bin:/home/htc/bzfernsa/swig-4.1.1:/home/htc/bzfernsa/swig/bin:/home/htc/bzfernsa/miniconda3/envs/Tolu39_2/bin/

Mind Tolu39_2 vs Tolu39_4.

/home/htc/bzfernsa/swig/bin:/home/htc/bzfernsa/miniconda3/envs/${whichever is the right one}/bin/ should occur rather early in PATH, as in the first output. But again, activating your python environment should have handled that correctly for you.

dweindl commented 1 year ago

Assuming the issue has been resolved. Please re-open if needed.