search

cdanielmachado / carveme

CarveMe: genome-scale metabolic model reconstruction
Other
151 stars 52 forks source link

[Feature request] Option to use non-proprietary solvers #173

Closed jolespin closed 10 months ago

jolespin commented 1 year ago

I'm testing it out on a prokaryotic genome and exceeded the limits for both cplex and gurobi. Are there any alternatives that are open source and non restrictive that could be used in its stead?

https://www.quora.com/What-are-some-good-open-source-alternatives-to-CPLEX-linear-program-solver

https://stackoverflow.com/questions/502102/best-open-source-mixed-integer-optimization-solver

https://en.wikipedia.org/wiki/List_of_optimization_software

I'd like to integrate this into my github.com/jolespin/veba software package but currently am limited by the dependencies.

cdanielmachado commented 1 year ago

We are currently working on finding an open source alternative to CPLEX.

For the moment, all users should apply for the IBM free academic license, which supports unlimited number of variables.

jolespin commented 1 year ago

Would this work?
https://anaconda.org/conda-forge/ortools-python https://github.com/google/or-tools https://developers.google.com/optimization/

About OR-Tools OR-Tools is an open source software suite for optimization, tuned for tackling the world's toughest problems in vehicle routing, flows, integer and linear programming, and constraint programming.

After modeling your problem in the programming language of your choice, you can use any of a half dozen solvers to solve it: commercial solvers such as Gurobi or CPLEX, or open-source solvers such as SCIP, GLPK, or Google's GLOP and award-winning CP-SAT.

jolespin commented 1 year ago

@cdanielmachado just checking in about this to see if there have been any updates. Cheers.

cdanielmachado commented 1 year ago

I am currently working on making SCIP available. Maybe in a few weeks it will be working. Just need to check the quality of the solutions and the models... :)

jolespin commented 1 year ago

That's exciting. Looking forward to testing it out! I have quite a few prokaryotic and eukaryotic MAGs I can use for trying it out.

bryantdo commented 11 months ago

Excited about this as well :)

cdanielmachado commented 10 months ago

This is now finally available in version 1.6.0. You can now use the SCIP solver (more details in the documentation).

It will be slower than a commercial solver and it is running with a time limit of 10 minutes (returns a suboptimal solution if the time limit is reached).

Let me know how it works, looking forward for feedback :)

pbelmann commented 10 months ago

This is now finally available in version 1.6.0. You can now use the SCIP solver (more details in the documentation).

It will be slower than a commercial solver and it is running with a time limit of 10 minutes (returns a suboptimal solution if the time limit is reached).

Let me know how it works, looking forward for feedback :)

Since you say it will be a sub-optimal solution, do you suggest using a commercial solver instead, if possible?

cdanielmachado commented 10 months ago

I ran some benchmarks and, despite the sub-optimal solution, the models performed just as well as those generated with Gurobi or CPLEX.

I would still recommend a commercial solver just for the speed.

jolespin commented 10 months ago

Can you describe how to use the sub-optimal solver with 1.6.0? https://pypi.org/project/carveme/

cdanielmachado commented 10 months ago

Sure, just add --solver scip as additional argument during execution.

jolespin commented 10 months ago

I got the following error with a fresh pip install: 

(VEBA-metabolism_env) [jespinoz@exp-15-02 TestVEBA]$ carve -h
Traceback (most recent call last):
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/bin/carve", line 5, in <module>
    from carveme.cli.carve import main
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/carveme/cli/carve.py", line 3, in <module>
    from carveme.reconstruction.carving import carve_model, build_ensemble
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/carveme/reconstruction/carving.py", line 8, in <module>
    from reframed.solvers import solver_instance, SCIPSolver
ImportError: cannot import name 'SCIPSolver' from 'reframed.solvers' (/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/reframed/solvers/__init__.py)

After installing SCIP with mamba install -c conda-forge scip

I got the following error: 

(VEBA-metabolism_env) [jespinoz@login01 share]$ carve -h
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
Traceback (most recent call last):
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/__init__.py", line 130, in <module>
    from numpy.__config__ import show as show_config
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.12 from "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/bin/python3.12"
  * The NumPy version is: "1.26.0"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: PyCapsule_Import could not import module "datetime"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/bin/carve", line 5, in <module>
    from carveme.cli.carve import main
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/carveme/__init__.py", line 5, in <module>
    from reframed.solvers.solver import default_parameters, Parameter
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/reframed/__init__.py", line 13, in <module>
    from .core.transformation import make_irreversible, simplify, gpr_transform
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/reframed/core/transformation.py", line 5, in <module>
    from ..cobra.variability import blocked_reactions
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/reframed/cobra/variability.py", line 4, in <module>
    from .thermodynamics import llFBA
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/reframed/cobra/thermodynamics.py", line 5, in <module>
    import numpy as np
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.12/site-packages/numpy/__init__.py", line 135, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.
cdanielmachado commented 10 months ago

I'm surprised that you even managed to install SCIP in python 3.12. It requests python 3.9 when I install it.

Can you try doing a fresh install of all packages in a new conda environment with python 3.9?

jolespin commented 10 months ago

I'm getting this error now with OpenBLAS:

carve -h 12> tmp.txt
OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 22 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 40 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 41 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 42 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 43 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 44 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 46 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 47 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 48 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 49 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 50 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 51 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 53 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 54 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 55 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 56 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 57 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 58 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 59 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 60 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 61 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 62 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
OpenBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 2000 current, 2000 max
Traceback (most recent call last):
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/bin/carve", line 5, in <module>
    from carveme.cli.carve import main
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/carveme/__init__.py", line 5, in <module>
    from reframed.solvers.solver import default_parameters, Parameter
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/reframed/__init__.py", line 13, in <module>
    from .core.transformation import make_irreversible, simplify, gpr_transform
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/reframed/core/transformation.py", line 5, in <module>
    from ..cobra.variability import blocked_reactions
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/reframed/cobra/variability.py", line 4, in <module>
    from .thermodynamics import llFBA
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/reframed/cobra/thermodynamics.py", line 6, in <module>
    from scipy.linalg import svd
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/scipy/linalg/__init__.py", line 206, in <module>
    from ._misc import *
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/scipy/linalg/_misc.py", line 3, in <module>
    from .blas import get_blas_funcs
  File "/expanse/projects/jcl110/anaconda3/envs/VEBA-metabolism_env/lib/python3.9/site-packages/scipy/linalg/blas.py", line 213, in <module>
    from scipy.linalg import _fblas
KeyboardInterrupt