wlandau / crew

A distributed worker launcher
https://wlandau.github.io/crew/
Other
129 stars 4 forks source link

Memory logging for workers #178

Closed wlandau closed 1 month ago

wlandau commented 3 months ago

As of 32fc13a35526589c52be15e4c40b9f9c8d3eab47, I added native memory logging functionality for local processes: https://wlandau.github.io/crew/articles/introduction.html#resources. The easiest way to use this is to set the log_resources argument of the controller to a file path where you want the memory log to live.

We can accomplish a similar monitoring scheme for workers by calling ps::ps_memory_info() from inside a task (and maybe users should be able to opt out using controller$push()). controller$summary() could then keep track of the max memory used by a task on a worker, and targets pipelines could keep track of that in the existing _targets/meta/crew file.

I won't have time to work on this for a while, but I will get to it eventually.

wlandau commented 3 months ago

Of course, this only works if the tasks succeed and can report back memory usage, but it's much easier than trying to reimplement this on a plugin-by-plugin basis.

wlandau commented 3 months ago

Even better: crew_eval() could just print a semantic message that could be grepped from any log file in any plugin.

wlandau commented 3 months ago

Possible improvement: a C thread that prints memory statistics in the background every so often. Might even be a good replacement for the current local memory logging system.

wlandau commented 1 month ago

I plan to start working on this next week using https://github.com/wlandau/autometric.