Closed kieranrcampbell closed 4 years ago
LTLA
commented
4 years ago If you're using the latest basilisk, there have been many, many changes to get it to build properly and safely on the Bioconductor build system. It seems that the last change to cellassign was last year, and I poured in 100 commits this month alone, so have a re-read of the vignette.
You may also consider reading Bioconductor/Contributions#1318 for some context.
kieranrcampbell
commented
4 years ago Yup I'm working locally, had updated according to the vignette to use e.g. BasiliskEnvironment. Will have a look at the bioc commits
LTLA
commented
4 years ago Can you reproduce the error on the son.of.basilisk example? We use a lazy installation strategy to get around the binary problem, but now there's a lot more moving parts than before.
kieranrcampbell
commented
4 years ago Yes:
> setwd("~/ubc/cellassign-project/basilisk/inst/example/")
> devtools::test()
Loading son.of.basilisk
Testing son.of.basilisk
✓ | OK F W S | Context
⠏ | 0 | basicCollecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1
added / updated specs:
- python=3.7.4
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2019.11.28 | py37_0 156 KB
------------------------------------------------------------
Total: 156 KB
The following NEW packages will be INSTALLED:
ca-certificates pkgs/main/osx-64::ca-certificates-2019.11.27-0
certifi pkgs/main/osx-64::certifi-2019.11.28-py37_0
libcxx pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
libcxxabi pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
libedit pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
libffi pkgs/main/osx-64::libffi-3.2.1-h475c297_4
ncurses pkgs/main/osx-64::ncurses-6.1-h0a44026_1
openssl pkgs/main/osx-64::openssl-1.1.1d-h1de35cc_3
pip pkgs/main/osx-64::pip-20.0.2-py37_0
python pkgs/main/osx-64::python-3.7.4-h359304d_1
readline pkgs/main/osx-64::readline-7.0-h1de35cc_5
setuptools pkgs/main/osx-64::setuptools-45.1.0-py37_0
sqlite pkgs/main/osx-64::sqlite-3.30.1-ha441bb4_0
tk pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
wheel pkgs/main/osx-64::wheel-0.33.6-py37_0
xz pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3
Downloading and Extracting Packages
certifi-2019.11.28 | 156 KB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
# $ conda activate /Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1
#
# To deactivate an active environment, use
#
# $ conda deactivate
==> WARNING: A newer version of conda exists. <==
current version: 4.7.12
latest version: 4.8.1
Please update conda by running
$ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1
added / updated specs:
- pandas=0.24.1
- python-dateutil=2.7.1
- python=3.7.4
- pytz=2018.7
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2019.11.28 | py37_0 148 KB conda-forge
------------------------------------------------------------
Total: 148 KB
The following NEW packages will be INSTALLED:
libblas conda-forge/osx-64::libblas-3.8.0-14_openblas
libcblas conda-forge/osx-64::libcblas-3.8.0-14_openblas
libgfortran conda-forge/osx-64::libgfortran-4.0.0-2
liblapack conda-forge/osx-64::liblapack-3.8.0-14_openblas
libopenblas conda-forge/osx-64::libopenblas-0.3.7-h3d69b6c_7
llvm-openmp conda-forge/osx-64::llvm-openmp-9.0.1-h28b9765_2
numpy conda-forge/osx-64::numpy-1.17.5-py37hde6bac1_0
pandas conda-forge/osx-64::pandas-0.24.1-py37h0a44026_0
python-dateutil conda-forge/noarch::python-dateutil-2.7.1-py_0
pytz conda-forge/noarch::pytz-2018.7-py_0
six conda-forge/osx-64::six-1.14.0-py37_0
The following packages will be UPDATED:
ca-certificates pkgs/main::ca-certificates-2019.11.27~ --> conda-forge::ca-certificates-2019.11.28-hecc5488_0
The following packages will be SUPERSEDED by a higher-priority channel:
certifi pkgs/main --> conda-forge
openssl pkgs/main::openssl-1.1.1d-h1de35cc_3 --> conda-forge::openssl-1.1.1d-h0b31af3_0
Downloading and Extracting Packages
certifi-2019.11.28 | 148 KB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
==> WARNING: A newer version of conda exists. <==
current version: 4.7.12
latest version: 4.8.1
Please update conda by running
$ conda update -n base -c defaults conda
x | 0 1 | basic [19.6 s]
──────────────────────────────────────────────────────────────────────────────────────────
test-basic.R:7: error: internal test package installs correctly
Unable to find conda binary. Is Anaconda installed?
Backtrace:
1. son.of.basilisk::test() tests/testthat/test-basic.R:7:4
2. basilisk::basiliskStart(env1) R/test.R:14:4
3. basilisk::useBasiliskEnv(envpath, required = FALSE)
4. reticulate::use_condaenv(envpath, required = required)
5. reticulate::conda_list(conda)
6. reticulate::conda_binary(conda)
──────────────────────────────────────────────────────────────────────────────────────────
══ Results ═══════════════════════════════════════════════════════════════════════════════
Duration: 19.6 s
OK: 0
Failed: 1
Warnings: 0
Skipped: 0sessionInfo as above
LTLA
commented
4 years ago Hm. Works for me. What does a fresh R session do here:
son.of.basilisk::test()Same issue:
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
[Workspace loaded from ~/.RData]
> devtools::load_all("~/ubc/cellassign-project/basilisk/inst/example/")
Loading son.of.basilisk
> son.of.basilisk::test()
Error: Unable to find conda binary. Is Anaconda installed?
> HMMM. traceback()?
kieranrcampbell
commented
4 years ago This is what I was referring to in the original issue -- you can modify use_condaenv to solve this but then get a later error that is even more mysterious
> traceback()
7: stop("Unable to find conda binary. Is Anaconda installed?", call. = FALSE)
6: conda_binary(conda)
5: conda_list(conda)
4: use_condaenv(envpath, required = required)
3: useBasiliskEnv(envpath, required = FALSE)
2: basiliskStart(env1) at test.R#14
1: son.of.basilisk::test()Can you debug(useBasiliskEnv) and tell me what the value of envpath is before it errors?
kieranrcampbell
commented
4 years ago Browse[2]> envpath
[1] "/Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1"And I assume this exists... okay. Let's try deleting /Users/kierancampbell/Library/Caches/basilisk and seeing if that helps.
kieranrcampbell
commented
4 years ago No such luck...same error
LTLA
commented
4 years ago I am bemused. The error comes from conda_list, but use_condaenv() only gets to that point if:
condaenv <- condaenv_resolve(condaenv)
if (grepl("[/\\]", condaenv) && is_condaenv(condaenv)) {is not TRUE. If you're passing in a full file path, the first expression should be fine, which suggests that the second expression is FALSE. This means that condaenv does not contain a conda-meta directory, which is rather odd for a conda environment.
So... does your env1 contain a conda-meta directory? You might also try to debug(use_condaenv) to see what condaenv is actually what we expected to be.
kieranrcampbell
commented
4 years ago (base) kierancampbell@Kierans-MacBook-Pro:~$ ls /Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1
bin/ conda-meta/ include/ lib/ share/ ssl/use_condaenvOn entry:
Browse[2]> condaenv
[1] "/Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1"
Browse[2]> conda
[1] "auto"
Browse[2]> Can go all the way to debug(conda_binary) where find_conda() returns NULL
LTLA
commented
4 years ago Need to go to a meeting now, but can you figure out why use_condaenv is bypassing that if?
kieranrcampbell
commented
4 years ago Browse[2]> is_condaenv(condaenv)
[1] TRUE
Browse[2]> grepl("[/\\]", condaenv, fixed = TRUE)
[1] FALSE
Browse[2]> condaenv
[1] "/Users/kierancampbell/Library/Caches/basilisk/0.99.54/son.of.basilisk-0.99.0/basilisk/env1"Wait - why do you have fixed=TRUE? I don't have that in my code (reticulate 1.14).
Edit: Interesting. Seems like OSX binaries are STILL not available for 1.14, judging by the CRAN landing page. Try re-installing reticulate from source for the time being.
kieranrcampbell
commented
4 years ago Success! Different error with cellassign now but I'll try and debug...Thanks!
LTLA
commented
4 years ago I'll finish up by saying that debugging is often a lot easier if basiliskStart() is asked to use an environment that is already loaded as the session's Python, i.e., py_config() shows Python from the desired environment. This allows you to use debug() and other helpful things to see what is happening inside the R functions being called within basiliskStart(). In contrast, if you are using a different environment from the session Python, basiliskStart() has to create a new process, and debugging becomes much harder.
Of course, there is an entirely separate family of bugs that could arise when you use functions inside basiliskStart() that assume that global variables in the calling environment are visible within basiliskStart(). This may not be true if a process needs to be created.
Hi @LTLA
Trying to update cellassign to use the latest version of conda. I've started getting the error
Unable to find conda binary. Is Anaconda installed?both in cellassign and in the
son.of.a.basiliskexample package. I've traced it back to thein the conditional at line 50 of
useBasiliskEnv.R. Looking up the error message seems to be that reticulate can't locate the conda binary, so I can point it towards that by modifying that line towhich appears to work fine but then errors later with
which appears triggered by a
clusterCall:and my debugging breaks down. Any hints?