Output ELBO loss during calibrate

[djinnome]

In PR https://github.com/ciemss/pyciemss/pull/163 we discussed that it would be helpful for the user to view the convergence of the SVI algorithm.

Currently, calibrate prints out the loss every 25 iterations. https://github.com/ciemss/pyciemss/blob/5da3f8347d47a7871b7d0b18d766108bb6565fb1/src/pyciemss/PetriNetODE/interfaces.py#L107-L111

Joshua would like to capture this information either as an actual (optional) output, or as logging output so that it could be visualized as it is happening.

good convergence

bad convergence

WTF convergence

[djinnome]

Hi Joseph, I assigned this to you because you may have opinions on whether it makes more sense to send the ELBO output to a logfile (for potential early stopping) or as an optional output of the calibrate function

[JosephCottam]

This is case where I think we should do both:

• If TA3 is interested in intermediate/live feedback, I think this is a good use-case and would pursue the "logging" option.
• Returning it from the function is a useful piece for calibrating trust in the result.

From an implementation standpoint, if we are doing both, I woudl go about it like this:

1. Create a file dedicated to the convergence data.
2. Make that file name available some how so it can be monitored. (A RESTful way to do this would be to return a work ID as part of the response and have another query for "what is the log file associated with this work id". Its an extra round-trip, but it makes it easy to get at the file from more than one client.)
3. Periodically write to the file.
4. When work is done, return the file ID as part of the result.

This setup does require some persistent storage so that files outlive jobs, but it is usually simple to setup: add a file handle to existing print statements, thread that file handle through the calls.

(The convergence plots don't show for me...broken images. I think I remember the images from notebooks, though.)

[SamWitty]

Addressed by progress_hook in calibrate.