ropensci / targets

Function-oriented Make-like declarative workflows for R
https://docs.ropensci.org/targets/
Other
937 stars 75 forks source link

Out of memory messaging #1201

Closed stemangiola closed 10 months ago

stemangiola commented 10 months ago

I have noticed that the message below comes to me whenever I launch a resource that does not have enough memory for a target. ( Out-of-memory is the #1 failure I meet after I sort out a functioning pipeline. )

Is it fair that, if this message is (also) linked with out-of-memory, it would be worth mentioning, "This can be caused by insufficient memory [...]. Please check [...], please do[...], please debug [...]".

This would be quite important in case I use Targets to build public pipelines for third-party users, where virtually the only failure point would be the memory.

For example, from this

Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: {crew} worker 2 launched 5 times in a row without completing any tasks. 
  Either raise launch_max above 5 or troubleshoot your platform to figure out why {crew} workers are not booting up or connecting.

To something like this

Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: {crew} worker 2 launched 5 times in a row without completing any tasks. 
  Either raise launch_max above 5 or troubleshoot your platform to figure out why {crew} workers are not booting up or connecting. 
  This could be due to insufficient memory (RAM) of the worker/controller for this target XXX. 
  A possible solution is to increase the worker's memory.

Thanks a lot.

wlandau commented 10 months ago

The error message comes from crew. I feel https://github.com/wlandau/crew/commit/405de3bed519119369169ca5647434f890894d46 now addresses this in a fully general way.