using basilisk on GitHub Actions

[bschilder]

Hello,

I was really excited to come across basilisk recently, as it addresses many of the things I've been trying to find hacks around for the last several years.

In the process, I've made a sort of meta-package called echoconda that handles building and using conda envs via different methods:

• reticulate
• basilisk
• command line wrapper

When i use basilisk for conda environment creation from yaml files via the function yaml_to_env(method="basilisk"), everything works fine when i run this locally. However, it keeps failing on my GitHub Actions workflow during the devtools::check step. You can see a recent example here.

The core error message relates to conda not installing, which i also use basilisk for via install_conda(method="basilisk"). I include a check within the first step of yaml_to_env that tries to ensure that conda is installed (if it isn't already).

Do you have any tips for using basilisk via GHA? (both for installing conda and for building conda envs).

Many thanks, Brian

[LTLA]

I don't have any particular insights to give, but I didn't think there would be any GHA-specific issues with it. Perhaps you could look at some of these other repos to see how they set up their Actions:

• https://github.com/kevinrue/velociraptor
• https://github.com/alanocallaghan/snifter
• https://github.com/theislab/zellkonverter

I don't see anything particularly special there, but who knows.

Anyway, looking at the code on your current HEAD indicates that yaml_to_env calls install_conda with method = "reticulate"? I don't recall what happens when you try to create basilisk environments with a non-basilisk-managed conda, I think you would have to set some special environment variables to get that working properly.

[bschilder]

Thanks for the suggestions, @LTLA Looking through these other repos now and seeing if I can find any clues.

I initially tried using only method="basilisk" for all relevant functions, but when that produced an error on GHA I started trying various combinations of basilisk/reticulate at each step. None of them allow the GHA to run all the way through yet, but i did notice I seem to get a bit further with installing conda and building the conda env via reticulate: https://github.com/RajLabMSSM/echoconda/runs/6199826099?check_suite_focus=true#step:23:1

However, it still runs into an error when ( for some reason) it's unable to find the conda binary again.

I wonder if this has anything to do with the fact that the first time i request conda to be installed / build the conda env, it's within the vignette (which is rmarkdown). I know occasionally weird errors pop up when knitting rmarkdowns that don't come up when running the same code in the R console. Not sure if this is a case of that here.

[bschilder]

I've managed to make a little bit of progress on this, but keep running into some barriers.

You can see on my latest echoconda GHA run that all tests passed on MacOS, but not on Linux or WIndows. I'm not quite sure why that would be though

https://github.com/RajLabMSSM/echoconda/actions/runs/2402079330

It seems to happen pretty consistently at the step where a small conda env (yaml specified here) is being built with the function:

conda_env <- echoconda::yaml_to_env("echoR_mini")

Here's the error on Linux, for example:

Loading required package: sessioninfo
── R CMD build ─────────────────────────────────────────────────────────────────
* checking for file ‘.../DESCRIPTION’ ... OK
* preparing ‘echoconda’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘echoconda.Rmd’ using rmarkdown
sh: 5: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/activate.d/activate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file548020cb1dab.sh: 4: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file548020cb1dab.sh: 5: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/activate.d/activate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file54802b8f076b.sh: 4: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file54802b8f076b.sh: 5: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/activate.d/activate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file54807ae95dd2.sh: 4: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
/tmp/RtmpmLHbQW/file5[48](https://github.com/RajLabMSSM/echoconda/runs/6638913065?check_suite_focus=true#step:23:49)07ae95dd2.sh: 5: /github/home/.cache/R/basilisk/1.9.0/echoconda/0.99.6/echoR_mini/etc/conda/activate.d/activate-r-base.sh: [[: not found
Quitting from lines 28-29 (echoconda.Rmd) 
Error: processing vignette 'echoconda.Rmd' failed with diagnostics:
--- failed re-building ‘echoconda.Rmd’
SUMMARY: processing the following file failed:
  ‘echoconda.Rmd’
Error: Vignette re-building failed.
Execution halted
Error in proc$get_built_file() : Build process failed
Calls: <Anonymous> ... build_package -> with_envvar -> force -> <Anonymous>
Execution halted
Error: Process completed with exit code 1.

[bschilder]

Wondering if perhaps it has something to do with the fact that basilisk builds envs using exact versions. I think the versions that I specified were from a conda i had built locally on my Mac.

[LTLA]

The error messages suggest that the various conda scripts are not being executed with bash, given that sh doesn't know anything about [[. I have no idea how it got to that point, though; I've never encountered this before, on GHA or otherwise.

[bschilder]

that's odd, I think bash should be available on all the GHA virtual machines. I wonder if there's a way to specify which interpreter to use

[bschilder]

Just an update, this seems to still be occurring on GHA. https://github.com/RajLabMSSM/echoconda/actions/runs/3356428963/jobs/5561487540#step:23:43

sh: 5: /github/home/.cache/R/basilisk/1.9.12/echoconda/0.99.8/echoR_mini/etc/conda/activate.d/activate-r-base.sh: [[: not found

Haven't found a solution to this yet, but will keep trying.

[bschilder]

I'm coming back to this and trying to make reticulate/basilisk compatible with rworkflows: https://github.com/neurogenomics/rworkflows/issues/78

I think setting up miniconda / mamba via some officially distributed actions may be helpful here: https://github.com/neurogenomics/rworkflows/issues/78#issuecomment-1725053205

However, this does mean that the GHA way of installing miniconda / mamba may diverge from how users actually install it from within R (e.g. reticulate::install_miniconda()).

I'll keep you posted as I press on @LTLA

[LTLA]

Thanks for the update @bschilder. It may be worth looping in some other basilisk users who work on GitHub; @kevinrue @alanocallaghan @lazappi @vjcitn etc. Maybe they have some experience with running basilisk inside GHAs.