thouska / spotpy

A Statistical Parameter Optimization Tool
https://spotpy.readthedocs.io/en/latest/
MIT License
248 stars 150 forks source link

Solving #266 with a new multiprocessing design #268

Open philippkraft opened 3 years ago

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 912


Files with Coverage Reduction New Missed Lines %
spotpy/algorithms/_algorithm.py 1 86.51%
<!-- Total: 1 -->
Totals Coverage Status
Change from base Build 910: 0.1%
Covered Lines: 4079
Relevant Lines: 4816

💛 - Coveralls
thouska commented 3 years ago

Thank you for the pull request. I think this could work, in fact it does wotk for 'ram' databases. However, switiching dbformat to 'csv' result in an TypeError: cannot pickle". Good news is, we do not need the parallel mode to know anythin about the database.

In case we switch to multiprocessing (as proposed here), we would need to hand over the user defined spot_setup class to the ForEach class only. So far it somehow gets to whole _algorithm via self. We might also need some queues for collecting and storing the data.

In case we stay with pathos.multiprocessing, we would need to figure out, why the csv database remains empty (happens if I try to reproduce #266). This is a strange error to me, as normal .txt writing does not result in any errors under pathos.multiprocessing in spotpy (tests locally).

I guess, we can follow up on both ways, having multiprocessing for pickable models and pathos for not pickable things. Would be nice to get at least one of them to run.