Error initialising - `"Subprocess is busy or cannot start"`

[MiguelRodo]

Hi Kevin

I think that the fix to #1577 has not entirely worked, unfortunately.

This is the call stack leading up to the error:

21: source("renv/activate.R")
20: withVisible(eval(ei, envir))
19: eval(ei, envir)
18: eval(ei, envir)
17: local(...) at activate.R#2
16: eval.parent(substitute(eval(quote(expr), envir)))
15: eval(expr, p)
14: eval(expr, p)
13: eval(quote(...), new.env())
12: eval(quote(...), new.env())
11: renv_bootstrap_exec(project, libpath, version) at activate.R#1194
10: if (!renv_bootstrap_load(project, libpath, version))
      renv_bootstrap_run(version, libpath) at activate.R#1007
 9: renv::load(project) at activate.R#862
 8: renv_load_bioconductor(project, lockfile$Bioconductor)
 7: renv_bioconductor_init()
 6: renv_bioconductor_init_biocmanager(library)
 5: install("BiocManager", library = library)
 4: renv_pak_install(packages, libpaths, project)
 3: pak$pkg_install(pkg = packages, lib = lib, upgrade = TRUE)
 2: remote(function(...) get("pkg_install_make_plan", asNamespace("pak"))(...), 
        list(pkg = pkg, lib = lib, upgrade = upgrade, ask = ask, 
            start = start, dependencies = dependencies, loaded = loaded_packages(lib)))
 1: stop("Subprocess is busy or cannot start")
R version 4.3.1 (2023-06-16) -- "Beagle Scouts"

I've managed to create a minimal example using a GitHub Codespace, a renv lockfile that specifies the BioConductor version a well as renv from the main branch on GitHub, and pak enabled. To repeat it:

1. Go to https://www.github.com/MiguelRodo/TestRenv
2. Click the green Code button.
3. Click the + button (if you hover, it will show Create a codespace on main).
4. Wait for build (< 5 minutes).
5. Open an R terminal by typing radian in terminal (using the R shell directly does the same thing, but does not print the call stack)

The error should then just happen after a while (a bit slow because I haven't disabled the sandbox).

[kevinushey]

This sounds like an issue in pak, since:

1: stop("Subprocess is busy or cannot start")

is the error pak returns when it's not able to start its remote process. I'm not sure how to get more information on these failures, though 😞

[MiguelRodo]

Thanks for the insight! Should I post this as an issue on the pak GitHub page?

[kevinushey]

Are you able to reproduce the issue using pak independently from renv?

[MiguelRodo]

I just checked, and no. So, if I comment out source("renv/activate") in .Rprofile, then pak::pkg_install("BiocManager") works fine, but when I uncomment it, I get exactly the same error. So, it seems to be an error I can induce without using a renv function, but it's not something I can induce without renv being activated.

I don't know what you make of this, but since I could induce the error directly, I tried pak::pkg_install("bioc::BiocManager") or pak::pkg_install("bioc::flowCore")(with renv activated) and the error went away even though pak::pkg_install("BiocManager") or pak::pkg_install("flowCore") give the error. I don't think BiocManager is from BioConductor, but I also tried pak::pkg_install("dplyr") and I got the same error that then went away when I ran pak::pkg_install("bioc::dplyr").

I tried debugging pak::pkg_install (debugonce(pak::pkg_install)), but then the error went away as I stepped through!

This isn't anything to do with a shim or anything renv starts? I know renv overrides, in some sense, install.packages, so I'm not sure if that's also being done here.

NOTE: the bioc:: "fix" seems kind of flaky - it worked a few times (in different R sessions), but just failed now (i.e. the error still appeared) when I started a new session and ran pak::pkg_install("bioc::cowplot").

The main upshot of above seems to be that pak::pkg_install works without renv being activated but fails when renv is activated. pak isn't listed in the lockfile, so I presume renv is installing the latest version.

[gaborcsardi]

Where is renv installing pak from, and how?

[MiguelRodo]

• As to how, it is either that a) renv is using pak installed by the devcontainer feature r-rig or b) it's ignoring that installed package and reinstaling it.
  • I'm fairly confident, however, that it is b), as I think renv ignores ignores previous pak installations (which was partly why I had #1577 - renv wasn't finding pak installed by the devcontainer feature and so had to install it itself).
• As to where from, it is https://r-lib.github.io/.
  • Whether renv is using pak from the devcontainer feature or its own install, both use that URL (as I read the install.sh script of the devcontainer feature and renv's pak functions at renv/R/pak.R).

In case it helps you to browse, all the code is in a minimal example at https://www.github.com/MiguelRodo/testRenv.

So, once the container is built, I run radian and I immediately get the error in the first message here.

[gaborcsardi]

Thanks, I am travelling now, but will take a look soon, probably in 2 days.

[kevinushey]

For reference, here's how renv installs pak: https://github.com/rstudio/renv/blob/d690785f01311d09678f1c696dcb55129d70de29/R/pak.R#L62-L94

[gaborcsardi]

@kevinushey Where is the pak installation there exactly? renv_namespace_load() sounds like it only loads pak from an installed location.

[gaborcsardi]

@MiguelRodo I tried to reproduce this as you suggested, but I got this:

R Language Server (1641) started
Error in h(simpleError(msg, call)) : failed to install:
installation of renv failed
===========================
ERROR: failed to lock directory ‘/workspaces/TestRenv/renv/library/R-4.3/x86_64-pc-linux-gnu’ for modifying
Try removing ‘/workspaces/TestRenv/renv/library/R-4.3/x86_64-pc-linux-gnu/00LOCK-rstudio-renv-5fb52fe’
Calls: source ... bootstrap -> withCallingHandlers -> renv_bootstrap_install
In addition: Warning message:
In system2(R, args, stdout = TRUE, stderr = TRUE) :
  running command ''/opt/R/4.3.1/lib/R/bin/R' --vanilla CMD INSTALL --no-multiarch -l '/workspaces/TestRenv/renv/library/R-4.3/x86_64-pc-linux-gnu' '/tmp/RtmpRBEM8h/renv_5fb52fefb8d919f2f3d88623b341c020330ed116.tar.gz' 2>&1' had status 3
Execution halted

R Language Server (1641) exited with exit code 1

[MiguelRodo]

Hi @gaborcsardi . Yes, I just then run sudo rm -rf /workspaces/TestRenv/renv/library/R-4.3/x86_64-pc-linux-gnu/00LOCK-rstudio-renv-5fb52fe (I don't think sudo is needed, but I just put it there). I'm not exactly sure why it comes up and just proceed from there.

[MiguelRodo]

Hi again @gaborcsardi. I created a rocker branch at the repo testRenv that uses rocker/verse as the base image for the codespace. That error about 00LOCK-rstudio-renv does not appear, but I get the same pak error (Subprocess is busy or cannot start).

[MiguelRodo]

Btw, @gaborcsardi, I think that Kevin might have given the code for renv loading pak, but the code for renv installing pak is a bit higher up in that script (here). As I read those functions, renv_pak_init is what installs and attaches pak as follows:

• It calls renv_pak_stream, which returns the stream of pak (from "stable", "rc" or "devel") that has a minimum version greater than the required minimum version, 0.5.1.
• renv_pak_init_impl then installs pak from that stream, getting the URL using renv_pak_repos and using install (renv's function) to install pak to whatever path is returned by renv_libpaths_active().

[gaborcsardi]

I'm partially guessing, but is it possible, that renv is also activated in the subprocess, and then renv/activate.R starts installing dependencies in the subprocess, which starts another pak subprocess, which again activates renv, and so on?

We had a somewhat similar issue, that was fixed in renv 1.0.1: https://github.com/r-lib/actions/issues/749 This was w/o pak, but AFAICT in renv 1.0.1 starting R does not trigger the installation of dependencies any more.

So, while IDK what renv/activate.R does in the different renv versions, it seems like it is worth trying this with renv 1.0.1.

[MiguelRodo]

TLDR: Previous issue involving infinite loops of sub-process when installing pak was resolved by getting renv not to use pak to install pak (#1577). Using the latest version of renv does not resolve this issue. renv/activate.R does in fact install the pak and BiocManager dependencies when starting R for the first time in a new container (when pak is enabled and BiocManager is a renv.lock-specified dependency).

---

Well, my previous issue (#1577) was basically that - if pak weren't installed, renv would attempt to install it, but this would go on forever because the installation was performed by a sub-process that itself would run renv/activate.R and so try to install pak in another sub-process before it (the first sub-process) could install pak (here's Kevin's summary, https://github.com/rstudio/renv/issues/1577#issuecomment-1661073531, and here's the fix, https://github.com/rstudio/renv/pull/1620, which was just to install pak without pak).

Kevin's fix solved that problem, but then this one cropped up immediately afterwards.

So, in that rocker branch I upgraded to renv version 1.0.2.9000, built a new codespace and the same issue arose.

So, if I turn pak off (RENV_CONFIG_PAK_ENABLED=false in .Renviron), then starting R will set up renv, install pak and then install BiocManager. So it does install those two dependencies automatically (if you need BioConductor, at least). If I have pak on, then it sets up renv and installs pak, but I get that error about the sub-process when it comes to installing BiocManager. I assume that's because renv does not use pak to set up itself and switches pak off when it installs pak (after #1577), and then goes back to using pak again.

Worth noting that, if you started R without pak enabled, let it do the above (set up renv, and install pak and BioConductor), then turn pak on (in the R session, i.e. Sys.setenv("RENV_CONFIG_PAK_ENABLED" = "TRUE")) and then run renv::restore(), you get the same pak error ("Subprocess is busy or cannot start"). So you can get that error after the inital start of R, and not just via renv/activate.R but also through renv::restore(). Of course, that may be obviously going to happen if renv/activate.R is sourced by renv::restore() in a new R sub-process anyway, which I guess is not unlikely.

[MiguelRodo]

TLDR: Unlike in #1577, BioConductor dependency does not seem to be a necessary part of the issue. renv::restore() triggers the pak error but renv::install("dplyr") (for example) does not.

---

So, in #1577 you'll see that the call stack up until the error showed us that the recursive install of pak only happened when BiocManager had to be installed due to a BioConductor dependency (in renv.lock). So, my minimal example included the following three lines in renv.lock to force the BioConductor dependency:

  "Bioconductor": {
    "Version": "3.17"
  },

I created a new branch in testRenv that does not have those lines in renv.lock (branch rocker-no_bioc), and rebuilt the codespace. I then did not get the pak error. However, when I run renv::restore(), I do get the pak error. I do not get the error when I run renv::install("dplyr"), for example.

This says the following:

• Unlike #1577, the issue is not BioConductor-specific, but only is induced earlier when BioConductor is a dependency as renv/activate.R installs BiocManager.
• The issue crops up with renv::restore() and not renv::install().
  • This seems a bit flaky, though, as Kevin called @gaborcsardi in after those very weird errors involving renv::install(). But that was in a container when I did have a BioConductor dependency, so maybe something odd was happening there involving BioConductor. In any case, it's not reproduced in a more-minimal environment (when the pak error here is still produced).

[MiguelRodo]

So, I've narrowed it down a bit more. It seems like, for some reason, renv can't both install pak and restore/install using pak in the same "movement", where by "movement" I mean during R startup, during a single call to renv::restore() or during a single call to renv::install. During R startup, renv presumably calls renv::restore, so the first two movements are the same thing, I think.

To reproduce:

1. Create a codespace using the rocker-no_bioc-dep branch (pak enabled, BioConductor reference removed from lock file and single dependency tinytest)
2. Run R in the terminal.
3. Respond to restore request. Two cases arise:
   • a). Suppose you say yes. Then it will install pak, and then return the Subprocess is busy or cannot start error. You then are in the regular R terminal, run renv::restore(), it will not ask you to install pak and the restore runs successfully.
   • b). Suppose you say no. Then it will not install pak. Then, once in the regular R terminal, you can reproduce the error by either restoring the renv project library or installing some random package, and agreeing for pak to be installed. The first time you do either of these, pak will be installed and then the Subprocess is busy or cannot start error arises. You then run that same command, and it will restore or install without complaint.

Side note: I had to move away from the rocker-no_bioc branch as, when you try to restore when there are zero dependencies (now that BiocManager is not a dependency), renv hangs as it tries to get pak to install nothing (stepping through renv_pak_restore showed that - it passes a character vector with no length to pak$pkg_install; I submitted a very short pull request re that. It's an odd scenario, as I had to hack away the BioConductor dependency, but it was creating an additional error in the minimal example).

[brancengregory]

Hi, has there been any progress in identifying a solution to this bug? I am receiving the same errors. I'd be happy to dig into the source to try to assist if not.

[brancengregory]

Interestingly this error does not occur on the windows-latest action runner

[iwalmsley]

Getting the same error, very keen to use pak through renv to speed up restore times.

[kevinushey]

pak 0.6.0 was just released; you might have better luck if you use the development version of renv + pak? You can force an update of pak in renv with:

renv:::renv_pak_init(force = TRUE)

It might take some time for pak 0.6.0 to make its way to the r-lib repository, though.

[MiguelRodo]

Thanks, Kevin. That seems to just install pak v0.5.1, though:

I then ran renv:::renv_pak_init(stream = "devel", force = TRUE), and that installed v0.5.1.9999:

Is that v0.6.0? It seems to be the latest from GitHub. Or do I need to set the repos arg to get v0.6.0?

In any case, it didn't seem to do the trick, unfortunately. I started a fresh container, ran renv:::renv_pak_init to get v0.5.1.9999 as above, restarted R, ran renv::restore(), accepted the prompt and ran into the same error:

Oddly enough, immediately rerunning renv::restore() again (in the same R session) worked fine. So even though pak was installed in a previous R session, the sub-process error occurred first thing in a new session when running renv::restore().

[gaborcsardi]

0.5.1.9999 is essentially the same as 0.6.0, but I don't think it has anything that would fix this issue, sadly.

[MiguelRodo]

I submitted a pull request to help solve this (https://github.com/rstudio/renv/pull/1665). Since the error only occurs the first time you use pak to install, I used tryCatch to repeat the attempt a couple of times if it fails (that is, fails due to this subprocess error). It's worked in my test setup (open R and accept prompts to restore lockfile in a fresh Ubuntu container), and passed the renv CI tests. And I can't really say there's a noticeable increase in time to restore due to the errors (even if you're just restoring one package).

So, it's a bit hacky, but the source of the error seems obscure and you can't avoid the error in each new (Ubuntu) container.

[gaborcsardi]

So this is now fixed in dev pak. Linux binaries will be available within an hour, the rest tomorrow.

OTOH,

1. I am not sure how to tell renv to use dev pak, and
2. https://github.com/rstudio/renv/issues/1678 means that it is not great to use pak for renv projects, with P3M.

[kevinushey]

You can manually reinstall the development version of pak in an renv project with:

renv:::renv_pak_init(stream = "devel", force = TRUE)

[MiguelRodo]

Thanks, @gaborcsardi! @kevinushey, just noting that this isn't really complete unless that line setting the minimum pak version in renv/R/pak.R is changed to be 0.6.0.9000

[kevinushey]

We'll probably wait for pak 0.6.1 to be released before bumping the minimum required version in renv.

[kevinushey]

Re-opening as reminder to bump minimum-required version of pak.

[MiguelRodo]

pak 0.7.0 released: https://github.com/r-lib/pak/releases/tag/v0.7.0

[emilBeBri]

I get this too. on a fully updated arch linux with:

> sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: EndeavourOS

Matrix products: default
BLAS/LAPACK: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_lp64.so.2;  LAPACK version 3.10.1

locale:
 [1] LC_CTYPE=en_DK.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=da_DK.UTF-8        LC_COLLATE=en_DK.UTF-8
 [5] LC_MONETARY=da_DK.UTF-8    LC_MESSAGES=en_DK.UTF-8
 [7] LC_PAPER=da_DK.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=da_DK.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Copenhagen
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] pak_0.7.1

loaded via a namespace (and not attached):
[1] compiler_4.3.2

trying to install anything with pak, does not matter which I think.

EDIT: SOLVED - OK apparently I had to update stringi to get rid of libicui18n.so.72: cannot open shared object file: No such file or directory. Why that is I got no clue, because I included the 'vanilla' flag. but it seemed like pak had nothing to do with it. Leaving the comment here so others might be assisted by it.

[MiguelRodo]

Hi Kevin

Sorry to restart this, but the issue crops up when installing BiocManager. It has been happening for a long time, but I've not gotten round to creating the reprex and reopening the issue.

In a Docker image without access to the BiocManager package, one gets this error when starting R in a renv project:

R version 4.4.0 (2024-04-24) -- "Puppy Cup"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Error in remote(function(...) get("pkg_install_make_plan", asNamespace("pak"))(...),  : 
  Subprocess is busy or cannot start
Traceback (most recent calls last):
21: source("renv/activate.R")
20: withVisible(eval(ei, envir))
19: eval(ei, envir)
18: eval(ei, envir)
17: local(...) at activate.R#2
16: eval.parent(substitute(eval(quote(expr), envir)))
15: eval(expr, p)
14: eval(expr, p)
13: eval(quote(...), new.env())
12: eval(quote(...), new.env())
11: renv_bootstrap_exec(project, libpath, version) at activate.R#1216
10: if (!renv_bootstrap_load(project, libpath, version))
      renv_bootstrap_run(version, libpath) at activate.R#1049
 9: renv::load(project) at activate.R#904
 8: renv_load_bioconductor(project, lockfile$Bioconductor)
 7: renv_bioconductor_init()
 6: renv_bioconductor_init_biocmanager(library)
 5: install("BiocManager", library = library, prompt = FALSE)
 4: renv_pak_install(packages, libpaths, project)
 3: pak$pkg_install(pkg = packages, lib = lib, upgrade = TRUE)
 2: remote(function(...) get("pkg_install_make_plan", asNamespace("pak"))(...), 
        list(pkg = pkg, lib = lib, upgrade = upgrade, ask = ask, 
            start = start, dependencies = dependencies, loaded = loaded_packages(lib)))
 1: stop("Subprocess is busy or cannot start")

The workaround is to run:

 Sys.setenv("RENV_CONFIG_PAK_ENABLED" = "false")
 renv::install("BiocManager")
 Sys.setenv("RENV_CONFIG_PAK_ENABLED" = "true")
 renv::restore()

To reproduce the issue, you can open the codespace from https://github.com/MiguelRodo/RenvExample and start R (note: I've edited it since re-opening the issue to remove unrelated files, as I realised I hadn't committed all the removals; no effect on bug). The lockfile only mentions BiocManager and one other small package.

Note that if renv is not called to install BiocManager, the error is not triggered. This can be seen from starting a codespace from the no_bioc branch of the MiguelRodo/RenvExample repo.

Could this issue be fixed by simply adding

renv_scope_options(renv.config.pak.enabled = FALSE)

to the renv_bioconductor_init_biocmanager function in the R/bioconductor.R file?

[kevinushey]

Sorry for the delay; I hope to investigate further next time I'm working on renv.