Periodically call arbitrary callback

[jbweston]

Currently adaptive-scheduler is capable of periodically saving the learner data, and calling 'learner.to_dataframe' and saving the resultant dataframe.

I would like to request the option to be able to additionally call an arbitrary callback. My use-case is that I would like to save the learner data in a more efficient format.

Failing this I could potentially override the 'save' method for my learners; does adaptive-scheduler assume anything about the filenames/data format produced by 'save'?

[basnijholt]

[This should be pretty easy to implement, because adaptive already accepts callbacks. We just need to pass the method and call it somewhere around here

https://github.com/basnijholt/adaptive-scheduler/blob/fee0703206b11300eddc9d241fbe55f4d0490eb9/adaptive_scheduler/_server_support/launcher.py#L145

Currently AFK, but will look tomorrow.

[jbweston]

Yes; I think the issue is piping 'save_method' all the way from the top of the API. We would also need to store 'save_method' somewhere so that the clients can pick it up.

Contrast this to 'save_dataframe', where we hardcode the method invocation and instead pass through simple data (boolean flag and a format specifier string).

[basnijholt]

We can probably pass it in the same way as the runner_kwargs: https://github.com/basnijholt/adaptive-scheduler/blob/fee0703206b11300eddc9d241fbe55f4d0490eb9/adaptive_scheduler/_server_support/launcher.py#L105

https://github.com/basnijholt/adaptive-scheduler/blob/fee0703206b11300eddc9d241fbe55f4d0490eb9/adaptive_scheduler/_server_support/job_manager.py#L73

[basnijholt]

@jbweston, I opened https://github.com/basnijholt/adaptive-scheduler/pull/209.