search

ICB-DCM / pyPESTO

python Parameter EStimation TOolbox
https://pypesto.readthedocs.io
BSD 3-Clause "New" or "Revised" License
219 stars 47 forks source link

Using SacessOptimizer on objectives with parameter priors leads to pickle error #1465

Closed stephanmg closed 1 week ago

stephanmg commented 1 month ago

Bug description Using a benchmark model with parameter priors leads to pickle error, see log below.

Expected behavior No pickle error and parameter estimation should start.

To reproduce Pick for instance the Schwen_PONE2014 benchmark model try to load and estimate ~with scipy or~ sacess optimizer.

"Remedy": Remove parameter priors from parameters.tsv works, but obviously then runs parameter estimation without prior information which is not desirable.

Latest confirmed working version of pyPESTO was 0.4.0 for me, might be the starting point of a git bisect.

Environment

Log:

main()
Traceback (most recent call last):
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 303, in <module>
    main()
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 286, in main
    problem, result = run_optimization(
                      ^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 112, in run_optimization
    result = optimizer.minimize(
             ^^^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/myvenv/lib/python3.11/site-packages/pypesto/optimize/ess/sacess.py", line 253, in minimize
    p.start()
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_prior_densities.<locals>.log_f'
Traceback (most recent call last):
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 303, in <module>
    main()
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 286, in main
    problem, result = run_optimization(
                      ^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 112, in run_optimization
    result = optimizer.minimize(
             ^^^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/myvenv/lib/python3.11/site-packages/pypesto/optimize/ess/sacess.py", line 253, in minimize
    p.start()
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_prior_densities.<locals>.log_f'
dweindl commented 1 month ago

This is specifically a problem with SacessOptimizer.

This problem was probably introduced in https://github.com/ICB-DCM/pyPESTO/pull/1353 (pypesto 0.5.0).

As a quick workaround, you can try using SacessOptimizer(..., mp_start_method="fork") if supported in your setup. This might potentially cause other issues, but it worked for me in a simple test.

For a proper fix, SacessOptimizer needs to use cloudpickle instead of pickle, as is already the case in MultiProcessEngine.

stephanmg commented 1 month ago

Workaround works for me.