search

usgs / pestpp

tools for scalable and non-intrusive parameter estimation, uncertainty analysis and sensitivity analysis
133 stars 71 forks source link

Frozen agents should be closed after successful completion of the manager #267

Open BJeannot1 opened 1 year ago

BJeannot1 commented 1 year ago

Hello,

I run PESTPP-IES on a supercomputer and automate my postprocessings to be executed right after the succesful completion of the ensemble smoother.

However, I found today that the cores allocated to my frozen agents are not freed after succesful completion of PESTPP-IES. This is a problem because my postprocessing won't start as long as all the cores have not been freed.

Do you think this issue is relevant ? If yes, could you please make it so that frozen agents are freed after the manager finished its work succesfully ?

Thank you for your help. Best regards, Benjamin JEANNOT

jtwhite79 commented 1 year ago

Hey @BJeannot1 . I think we could make workers exit from a frozen state when the master exits...but that freezing is really a debugging option and I'm worried unfreezing when the master exits might disrupt that debugging ability, esp in managed envs that are cleaned automatically - I don't leave that option "on", but only activate it when I'm chasing issues. If you do want records from failed runs, maybe you use the "transfer on fail" options instead to bring back specific files from run fails? Does that make sense?