search

DOI-BOR / ostrich

An optimization toolkit for model calibration
GNU General Public License v2.0
1 stars 2 forks source link

Wish List: Look for model failure indicators #41

Open analytophile opened 2 years ago

analytophile commented 2 years ago

It would be nice to be able to look for a keyword in a log file that indicates when a model has failed and stop optimization. If the command is not present, nothing would be looked for, i.e., standard operation

Snowthe commented 2 years ago

I understand the wish, but doesn't this easily become complex, if it is supposed to be useful?

In the easiest case, this log file is a text file and you simply have to look for a specific string. So far so good. But, dependent on your model and task at hand, you might want to handle a failed model run differently (run the model again? pass "penalty" to OSTRICH? Stop OSTRICH?). I handle such cases in my model executable instead (i.e. a batch file).

Also, OSTRICH already offers the parameter "OnObsError" that allows to specify what happens in case OSTRICH cannot read an observation.

analytophile commented 2 years ago

I was envision the easiest case - it stops Ostrich if the model is problematic rather than running 5000 broken models. The logic for when the model is sufficiently problematic to shut it down is still left up to the model or batch file.

Its unclear to me how you would get Ostrich to stop from a batch file, but I'm willing to learn.

The OnObsError is a distinct functionality - it says nothing about the model.

dloney commented 2 years ago

The onobserror should do this, if I'm understanding correctly. As long as there's not a placeholder value where OSTRICH is trying to do the read, if it's set to "quit" it should fail on an improperread. A couple ways of accomplishing it would be to either make sure your solve files from the previous iteration are deleted so there's no data to create a bad read or have another script that adjusts the output files after checking for proper completion.

There's not a direct way currently for OSTRICH to check if a call has properly executed. Since most workflows are multi-part, I'd have to think if there's a way to handle a call failure gracefully across platforms.