search

ICB-DCM / pyABC

distributed, likelihood-free inference
https://pyabc.rtfd.io
BSD 3-Clause "New" or "Revised" License
205 stars 44 forks source link

PendingRollbackError reported when using pyabc #569

Closed xieduo7 closed 2 years ago

xieduo7 commented 2 years ago

Dear @yannikschaelte,

The script reported an error when I ran the pyABC script. Do you have any idea about this error? Is this caused by the SQL storage?

Thank you!

ABC.Sampler INFO: Parallelize sampling on 80 processes.
ABC.History INFO: Start <ABCSMC id=3, start_time=2022-05-05 17:05:58>
ABC INFO: Calibration sample t = -1.
ABC INFO: t: 0, eps: 6.56922806e-02.
 PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (builtins.ValueError) Object arrays cannot be saved when allow_pickle=False
[SQL: INSERT INTO summary_statistics (sample_id, name, value) VALUES (?, ?, ?)]
[parameters: [{'name': 'X_2', 'sample_id': 4, 'value': array([0.2358974358974359, 0.11904761904761904, 0.09268292682926829, ...,
       0.5618448637316562, 0.3875968992248062, 0.43537414965986393],
      dtype=object)}]] (Background on this error at: https://sqlalche.me/e/14/7s2a) 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1707 
-> 1708             context = constructor(
   1709                 dialect, self, conn, execution_options, *args, **kw

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py in _init_compiled(cls, dialect, connection, dbapi_connection, execution_options, compiled, parameters, invoked_statement, extracted_parameters, cache_hit)
   1069             for compiled_params in self.compiled_parameters:
-> 1070                 param = [
   1071                     processors[key](compiled_params[key])

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py in <listcomp>(.0)
   1070                 param = [
-> 1071                     processors[key](compiled_params[key])
   1072                     if key in processors

~/.local/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py in process(value)
   1630                 def process(value):
-> 1631                     return impl_processor(process_param(value, dialect))
   1632 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/db_model.py in process_bind_param(self, value, dialect)
     39     def process_bind_param(self, value, dialect):  # pylint: disable=R0201
---> 40         return to_bytes(value)
     41 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/bytes_storage.py in to_bytes(object_)
     37         return df_to_bytes(object_)
---> 38     return np_to_bytes(object_)
     39 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/numpy_bytes_storage.py in np_to_bytes(arr)
     21     f = BytesIO()
---> 22     np.save(f, arr, allow_pickle=False)
     23     f.seek(0)

<__array_function__ internals> in save(*args, **kwargs)

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
    527         arr = np.asanyarray(arr)
--> 528         format.write_array(fid, arr, allow_pickle=allow_pickle,
    529                            pickle_kwargs=dict(fix_imports=fix_imports))

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    659         if not allow_pickle:
--> 660             raise ValueError("Object arrays cannot be saved when "
    661                              "allow_pickle=False")

ValueError: Object arrays cannot be saved when allow_pickle=False

The above exception was the direct cause of the following exception:

StatementError                            Traceback (most recent call last)
~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/inference/smc.py in wrapped_run(self, *args, **kwargs)
     61             # the actual run
---> 62             ret = run(self, *args, **kwargs)
     63         finally:

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/inference/smc.py in run(self, minimum_epsilon, max_nr_populations, min_acceptance_rate, max_total_nr_simulations, max_walltime, min_eps_diff)
    697             # perform one generation
--> 698             ret = self.run_generation(t=t)
    699 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/inference/smc.py in run_generation(self, t)
    837         model_names = [model.name for model in self.models]
--> 838         self.history.append_population(
    839             t, current_eps, population, n_sim, model_names

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/history.py in append_population(self, t, current_epsilon, population, nr_simulations, model_names)
    810 
--> 811         self._save_to_population_db(
    812             t,

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/history.py in f_wrapper(self, *args, **kwargs)
     41             self._make_session()
---> 42         res = f(self, *args, **kwargs)
     43         if no_session:

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/history.py in _save_to_population_db(self, t, current_epsilon, nr_simulations, particles_by_model, model_probabilities, model_names)
    777         # commit changes
--> 778         self._session.commit()
    779 

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in commit(self)
   1430 
-> 1431         self._transaction.commit(_to_root=self.future)
   1432 

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in commit(self, _to_root)
    828         if self._state is not PREPARED:
--> 829             self._prepare_impl()
    830 

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _prepare_impl(self)
    807                     break
--> 808                 self.session.flush()
    809             else:

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in flush(self, objects)
   3362             self._flushing = True
-> 3363             self._flush(objects)
   3364         finally:

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _flush(self, objects)
   3502             with util.safe_reraise():
-> 3503                 transaction.rollback(_capture_exception=True)
   3504 

~/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py in __exit__(self, type_, value, traceback)
     69             if not self.warn_only:
---> 70                 compat.raise_(
     71                     exc_value,

~/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py in raise_(***failed resolving arguments***)
    206         try:
--> 207             raise exception
    208         finally:

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _flush(self, objects)
   3462             try:
-> 3463                 flush_context.execute()
   3464             finally:

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py in execute(self)
    455             for rec in topological.sort(self.dependencies, postsort_actions):
--> 456                 rec.execute(self)
    457 

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py in execute(self, uow)
    629     def execute(self, uow):
--> 630         util.preloaded.orm_persistence.save_obj(
    631             self.mapper,

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py in save_obj(base_mapper, states, uowtransaction, single)
    243 
--> 244         _emit_insert_statements(
    245             base_mapper,

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py in _emit_insert_statements(base_mapper, uowtransaction, mapper, table, insert, bookkeeping)
   1236                     else:
-> 1237                         result = connection._execute_20(
   1238                             statement,

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _execute_20(self, statement, parameters, execution_options)
   1619         else:
-> 1620             return meth(self, args_10style, kwargs_10style, execution_options)
   1621 

~/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py in _execute_on_connection(self, connection, multiparams, params, execution_options, _force)
    324         if _force or self.supports_execution:
--> 325             return connection._execute_clauseelement(
    326                 self, multiparams, params, execution_options

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _execute_clauseelement(self, elem, multiparams, params, execution_options)
   1486         )
-> 1487         ret = self._execute_context(
   1488             dialect,

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1713         except BaseException as e:
-> 1714             self._handle_dbapi_exception(
   1715                 e, util.text_type(statement), parameters, None, None

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   2031             elif should_wrap:
-> 2032                 util.raise_(
   2033                     sqlalchemy_exception, with_traceback=exc_info[2], from_=e

~/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py in raise_(***failed resolving arguments***)
    206         try:
--> 207             raise exception
    208         finally:

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1707 
-> 1708             context = constructor(
   1709                 dialect, self, conn, execution_options, *args, **kw

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py in _init_compiled(cls, dialect, connection, dbapi_connection, execution_options, compiled, parameters, invoked_statement, extracted_parameters, cache_hit)
   1069             for compiled_params in self.compiled_parameters:
-> 1070                 param = [
   1071                     processors[key](compiled_params[key])

~/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py in <listcomp>(.0)
   1070                 param = [
-> 1071                     processors[key](compiled_params[key])
   1072                     if key in processors

~/.local/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py in process(value)
   1630                 def process(value):
-> 1631                     return impl_processor(process_param(value, dialect))
   1632 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/db_model.py in process_bind_param(self, value, dialect)
     39     def process_bind_param(self, value, dialect):  # pylint: disable=R0201
---> 40         return to_bytes(value)
     41 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/bytes_storage.py in to_bytes(object_)
     37         return df_to_bytes(object_)
---> 38     return np_to_bytes(object_)
     39 

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/numpy_bytes_storage.py in np_to_bytes(arr)
     21     f = BytesIO()
---> 22     np.save(f, arr, allow_pickle=False)
     23     f.seek(0)

<__array_function__ internals> in save(*args, **kwargs)

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
    527         arr = np.asanyarray(arr)
--> 528         format.write_array(fid, arr, allow_pickle=allow_pickle,
    529                            pickle_kwargs=dict(fix_imports=fix_imports))

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    659         if not allow_pickle:
--> 660             raise ValueError("Object arrays cannot be saved when "
    661                              "allow_pickle=False")

StatementError: (builtins.ValueError) Object arrays cannot be saved when allow_pickle=False
[SQL: INSERT INTO summary_statistics (sample_id, name, value) VALUES (?, ?, ?)]
[parameters: [{'name': 'X_2', 'sample_id': 4, 'value': array([0.2358974358974359, 0.11904761904761904, 0.09268292682926829, ...,
       0.5618448637316562, 0.3875968992248062, 0.43537414965986393],
      dtype=object)}]]

During handling of the above exception, another exception occurred:

PendingRollbackError                      Traceback (most recent call last)
<ipython-input-1-099cacb0f3b4> in <module>
    368 abc.new(db_path, {"X_2": sum_stat_obs})
    369 #history = abc.run(minimum_epsilon=5, max_nr_populations=10)
--> 370 history_3model_improved_2sub_5000_15_multi_5 = abc.run(max_nr_populations=3)

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/inference/smc.py in wrapped_run(self, *args, **kwargs)
     63         finally:
     64             # close session and store end time
---> 65             self.history.done(end_time=datetime.now())
     66             # tell samplers to stop any ongoing processes
     67             self.sampler.stop()

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/history.py in f_wrapper(self, *args, **kwargs)
     40         if no_session:
     41             self._make_session()
---> 42         res = f(self, *args, **kwargs)
     43         if no_session:
     44             self._close_session()

~/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages/pyabc/storage/history.py in done(self, end_time)
    689             end_time = datetime.datetime.now()
    690 
--> 691         abcsmc = self._session.query(ABCSMC).filter(ABCSMC.id == self.id).one()
    692         abcsmc.end_time = end_time
    693         self._session.commit()

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py in one(self)
   2854 
   2855         """
-> 2856         return self._iter().one()
   2857 
   2858     def scalar(self):

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py in _iter(self)
   2892 
   2893         statement = self._statement_20()
-> 2894         result = self.session.execute(
   2895             statement,
   2896             params,

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in execute(self, statement, params, execution_options, bind_arguments, _parent_execute_state, _add_event, **kw)
   1689             )
   1690         else:
-> 1691             conn = self._connection_for_bind(bind)
   1692         result = conn._execute_20(statement, params or {}, execution_options)
   1693 

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _connection_for_bind(self, engine, execution_options, **kw)
   1530 
   1531         if self._transaction is not None or self._autobegin():
-> 1532             return self._transaction._connection_for_bind(
   1533                 engine, execution_options
   1534             )

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _connection_for_bind(self, bind, execution_options)
    719 
    720     def _connection_for_bind(self, bind, execution_options):
--> 721         self._assert_active()
    722 
    723         if bind in self._connections:

~/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py in _assert_active(self, prepared_ok, rollback_ok, deactive_ok, closed_msg)
    599             if not deactive_ok and not rollback_ok:
    600                 if self._rollback_exception:
--> 601                     raise sa_exc.PendingRollbackError(
    602                         "This Session's transaction has been rolled back "
    603                         "due to a previous exception during flush."

PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (builtins.ValueError) Object arrays cannot be saved when allow_pickle=False
[SQL: INSERT INTO summary_statistics (sample_id, name, value) VALUES (?, ?, ?)]
[parameters: [{'name': 'X_2', 'sample_id': 4, 'value': array([0.2358974358974359, 0.11904761904761904, 0.09268292682926829, ...,
       0.5618448637316562, 0.3875968992248062, 0.43537414965986393],
      dtype=object)}]] (Background on this error at: https://sqlalche.me/e/14/7s2a)

Best, Duo

yannikschaelte commented 2 years ago

Hi @xieduo7 , can you please share the system and version info as described here, and report the sqlalchemy version?

xieduo7 commented 2 years ago

Hi @yannikschaelte,

Thank you for your quick response and sorry for not sharing this information.

Environment [Report at least the pyABC version (pip show pyabc),

pip show pyabc
Name: pyabc
Version: 0.12.3
Summary: Distributed, likelihood-free ABC-SMC inference
Home-page: https://github.com/icb-dcm/pyabc
Author: The pyABC developers
Author-email: yannik.schaelte@gmail.com
License: BSD-3-Clause
Location: /home/xieduo/software/miniconda3/envs/jupyter-env/lib/python3.8/site-packages
Requires: scikit-learn, jabbar, redis, pandas, matplotlib, cloudpickle, click, sqlalchemy, scipy, distributed, numpy, gitpython
Required-by:

python version (which python; python --version), Python 3.8.13

and system (Ubuntu/Mac/...).] Linux pbs-master 3.10.0-1160.41.1.el7.x86_64 #1 SMP Tue Aug 31 14:52:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

sqlalchemy version

$ pip show sqlalchemy
Name: SQLAlchemy
Version: 1.4.32
Summary: Database Abstraction Library
Home-page: https://www.sqlalchemy.org
Author: Mike Bayer
Author-email: mike_mp@zzzcomputing.com
License: MIT
Location: /home/xieduo/.local/lib/python3.8/site-packages
Requires: greenlet
Required-by: pyabc

Thank you.

Best, Duo

yannikschaelte commented 2 years ago

Hi, sorry for not answering, this issue went amiss. It appears you have a ValueError: Object arrays cannot be saved when allow_pickle=False, which can indicate that your summary statistics contain values different from e.g. floats, strings, ints etc, which cannot be correctly stored, see https://pyabc.readthedocs.io/en/latest/api_datastore.html#what-can-be-stored-as-summary-statistics

yannikschaelte commented 2 years ago

closing due to non-response. feel free to re-open if issue persists.