It's an on-going problem that we're seeing where bgcval2 is running incredibly slowly, even on lotus and clusters. (@DrYool will corroborate)
I think some parallelisation would be worth doing.
At the moment, we're running each job in series, but I'd like to farm them out to individual machines. It would be good to figure out the best way to do this. Inside ESMValTool seems like it would be inefficient, as you don't know how many jobs you'd want to analyse. (say you have 9 jobs, but you split it up 8 ways, you're going to be waiting for that last run to complete.
So here's my idea. Lets generate and submit a basic lotus script for each job, then we can send them to the queue and forget about them.
The script would be very similar to lotus_bgcval2.sh but runs analysis_timeseries instead of analysis_compare.
Thoughts, @valeriupredoi ?
Potential issues:
Need to be careful to make sure that the same analysis isn't running twice at the same time.
It's an on-going problem that we're seeing where bgcval2 is running incredibly slowly, even on lotus and clusters. (@DrYool will corroborate)
I think some parallelisation would be worth doing.
At the moment, we're running each job in series, but I'd like to farm them out to individual machines. It would be good to figure out the best way to do this. Inside ESMValTool seems like it would be inefficient, as you don't know how many jobs you'd want to analyse. (say you have 9 jobs, but you split it up 8 ways, you're going to be waiting for that last run to complete.
So here's my idea. Lets generate and submit a basic lotus script for each job, then we can send them to the queue and forget about them.
The script would be very similar to
lotus_bgcval2.shbut runs analysis_timeseries instead ofanalysis_compare.Thoughts, @valeriupredoi ?
Potential issues: