search

gem / oq-engine

OpenQuake Engine: a software for Seismic Hazard and Risk Analysis
https://github.com/gem/oq-engine/#openquake-engine
GNU Affero General Public License v3.0
380 stars 273 forks source link

Making sure atexit functions are called before shutting down the ProcessPool #9674

Closed micheles closed 4 months ago

micheles commented 4 months ago

Hopefully will fix the error in the oq-risk-tests

[gw1] linux -- Python 3.11.7 /home/openquake/openquake/bin/python3
worker 'gw1' crashed while running 'tests.py::test_conditioned_gmfs2'
/usr/local/lib/python3.11/multiprocessing/resource_tracker.py:254:
UserWarning: resource_tracker: There appear to be 2 leaked shared_memory
objects to clean up at shutdown
micheles commented 4 months ago

No, it caused another error because the datastores are closed too early by atexit:

tests.py:475: in test_conditioned_gmfs2
    gmf, sigeps, sites = export(('gmf_data', 'csv'), calc.datastore)
../oq-engine/openquake/baselib/general.py:668: in __call__
    return self[key](obj, *args, **kw)
../oq-engine/openquake/calculators/export/hazard.py:342: in export_gmf_data_csv
    oq = dstore['oqparam']
../oq-engine/openquake/commonlib/datastore.py:517: in __getitem__
    raise KeyError('No %r found in %s' % (key, self))
E   KeyError: "No 'oqparam' found in <DataStore /home/michele/oqdata/calc_65750.hdf5 open>"