`crew` package not available in background task launched by `crew`

yogat3ch commented 6 months ago

Prework

[x] Read and agree to the Contributor Code of Conduct and contributing guidelines.
[x] Confirm that your issue is a genuine bug in the crew package itself and not a user error, known limitation, or issue from another package that crew depends on. For example, if you get errors running tar_make_clustermq(), try isolating the problem in a reproducible example that runs clustermq and not crew. And for miscellaneous troubleshooting, please post to discussions instead of issues.
[x] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
[ ] Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
- [ ] Runnable: post enough R code and data so any onlooker can create the error on their own computer.
- [ ] Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
- [ ] Readable: format your code according to the tidyverse style guide.

Description

crew seems to be working fine in all of our AWS environments except for one in which very bizarre behavior is happening. The background tasks launched by crew give an error indicating that the crew package doesn't exist. Here's the logs written by crew that show this anomaly:

{

R version 4.3.0 (2023-04-21) -- "Already Tomorrow"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: aarch64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> crew::crew_worker(settings = list(url = "ws://127.0.0.1:37039/02fe7f0a3bb57842a2e17c32", autoexit = 15L, cleanup = 1L, output = TRUE, maxtasks = Inf, idletime = 10000, walltime = Inf, timerstart = 0L, tls = NULL, rs = c(10407L, 1461037220L, 2123128661L, -694900718L, 161841259L, -1585953616L, -1384386319L)), launcher = "email@email.com", worker = 1L, instance = "02fe7f0a3bb57842a2e17c32")
Error in loadNamespace(x) : there is no package called ‘crew’
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted,

Reproducible example

I'm unable to do so as it involves running in a remote arm64 linux container on AWS.

Expected result

On all of our other environments, crew seems to function. On this environment, on occasion, crew will cause intermittent errors with these logs. Have you encountered anything like this? How might crew launch a worker, and then launch a task that is unable to find the crew package on that worker when trying to run the task? Perhaps it's somehow loading with a different library path? I'm stumped.

Diagnostic information

R. 4.3.0 on aarch64-unknown-linux-gnu (64-bit)

yogat3ch commented 6 months ago

I think I know why this is now. We're using renv to restore all of the dependencies for our golem app. renv writes all the packages to it's own library directory, rather than the standard R library directory. I think the R session that launches in the background is likely using the default library path, and isn't inheriting from the parent session. Does the background R session loaded by mirai load the local .Rprofile as a typical R session? I could add code to the .Rprofile to solve this if so. I'm going to test that out

shikokuchuo commented 6 months ago

Yes, the background R sessions should read .Rprofile.

wlandau commented 6 months ago

You can also manually supply library paths to the library argument of controller$push(): https://wlandau.github.io/crew/reference/crew_class_controller.html#method-crew_class_controller-push

wlandau / crew