search

thouska / spotpy

A Statistical Parameter Optimization Tool
https://spotpy.readthedocs.io/en/latest/
MIT License
247 stars 149 forks source link

sceua & mpc : writing data in txt error for mpc #304

Open ZhiqiangD opened 1 year ago

ZhiqiangD commented 1 year ago

Hello everyone, I'm new to using spotpy and I have a question. I'm currently trying to use the sceua algorithm with mpc to calibrate VIC. The problem is that the parameters I need to write to a txt file for VIC to use are getting overlapped or mixed up during the parallel calibration process. I've tried adding process locks but it didn't work. Can anyone please suggest a solution or offer some advice? Thank you so much!

ZhiqiangD commented 1 year ago

Hi, in tutorial_parallel_computing_hymod.py,
If your model in def simulation reads any files, make sure, they are unique for each CPU. Otherwise, things get messed up... spot_setup = spot_setup(parallel=parallel)

Could you please advise me on how to ensure that each CPU uses a file that is unique to it, for example, by following the instructions in this link (https://github.com/thouska/spotpy/issues/266#issuecomment-848695555) or by using joblib (https://github.com/thouska/spotpy/issues/266#issuecomment-861491408)?

Thanks @thouska

thouska commented 1 year ago

Hi @ZhiqiangD, thank you for your message and for using spotpy for your research. And perfect that you found already the relevant information for you issue. For the multiprocessing spotpy is using pathos.multiprocessing. So, we just need to figure out, how to ask pathos, on which CPU it is currently running. This value can be used to make copies of your VIC model inputs. This link looks like it covers this point.

ZhiqiangD commented 1 year ago

Hi @thouska, I apologize for the delayed response. After conducting some experiments, I have successfully achieved 'mpc' calibration. However, I have encountered two issues with 'mpc'. Firstly, I am unable to control the number of CPU cores utilized by the program, as it automatically utilizes all available CPU cores on my computer, even when I set ngs=4 or ngs=8. Secondly, after completing multi-core runs, the results in the database still show the number of rate determinations rather than the rate determination times the number of CPU cores used. I was wondering if the algorithm automatically selects the best result from each CPU folder and includes it in the final database. Thank you for your assistance in advance!