snifter - Githubissues

alanocallaghan commented 4 years ago

Update the following URL to point to the GitHub repository of the package you wish to submit to Bioconductor

Repository: https://github.com/Alanocallaghan/snifter

Confirm the following by editing each check box to '[x]'

[x] I understand that by submitting my package to Bioconductor, the package source and all review commentary are visible to the general public.
[x] I have read the Bioconductor Package Submission instructions. My package is consistent with the Bioconductor Package Guidelines.
[x] I understand that a minimum requirement for package acceptance is to pass R CMD check and R CMD BiocCheck with no ERROR or WARNINGS. Passing these checks does not result in automatic acceptance. The package will then undergo a formal review and recommendations for acceptance regarding other Bioconductor standards will be addressed.
[x] My package addresses statistical or bioinformatic issues related to the analysis and comprehension of high throughput genomic data.
[x] I am committed to the long-term maintenance of my package. This includes monitoring the support site for issues that users may have, subscribing to the bioc-devel mailing list to stay aware of developments in the Bioconductor community, responding promptly to requests for updates from the Core team in response to changes in R or underlying software.

I am familiar with the essential aspects of Bioconductor software management, including:

[x] The 'devel' branch for new packages and features.
[x] The stable 'release' branch, made available every six months, for bug fixes.
[x] Bioconductor version control using Git (optionally via GitHub).

For help with submitting your package, please subscribe and post questions to the bioc-devel mailing list.

bioc-issue-bot commented 4 years ago

Hi @Alanocallaghan

Thanks for submitting your package. We are taking a quick look at it and you will hear back from us soon.

The DESCRIPTION file for this package is:

Package: snifter
Type: Package
Title: R wrapper for the python openTSNE library
Version: 0.99.0
Date: 2020-07-17
Authors@R: c(
    person("Alan", "O'Callaghan", role = c("aut", "cre"), email = "alan.ocallaghan@outlook.com"),
    person("Aaron", "Lun", role = c("aut")))
Depends: R (>= 4.0.0)
Imports: 
    basilisk,
    reticulate,
    assertthat
Description:
    Provides an R wrapper for the implementation of FI-tSNE from the
    python package openTNSE. See Poličar et al. (2019) <doi:10.1101/731877>
    and the algorithm described by Linderman et al. (2018)
    <doi:10.1038/s41592-018-0308-4>.
License: GPL-3
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.1.1
VignetteBuilder: knitr
biocViews: DimensionReduction, Visualization, Software, SingleCell, Sequencing
StagedInstall: no
BugReports: https://github.com/Alanocallaghan/snifter/issues
Suggests: 
    knitr,
    rmarkdown,
    scRNAseq,
    BiocStyle,
    scater,
    scran,
    scuttle,
    testthat

bioc-issue-bot commented 4 years ago

A reviewer has been assigned to your package. Learn what to expect during the review process.

IMPORTANT: Please read this documentation for setting up remotes to push to git.bioconductor.org. It is required to push a version bump to git.bioconductor.org to trigger a new build.

Bioconductor utilized your github ssh-keys for git.bioconductor.org access. To manage keys and future access you may want to active your Bioconductor Git Credentials Account

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "ERROR, skipped". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details. This link will be active for 21 days.

Remember: if you submitted your package after July 7th, 2020, when making changes to your repository push to git@git.bioconductor.org:packages/snifter to trigger a new build. A quick tutorial for setting up remotes and pushing to upstream can be found here.

mtmorgan commented 4 years ago

is this a complicated conda / python configuration issue on the build system @LTLA @hpages http://bioconductor.org/spb_reports/snifter_buildreport_20200721160231.html#merida1_buildsrc_anchor

alanocallaghan commented 4 years ago

I'm told this is relevant https://github.com/Bioconductor/Contributions/issues/1564#issuecomment-661966974

mtmorgan commented 4 years ago

@Alanocallaghan do you mean that the error reported there

ERROR: R installation problem: File /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libgfortran.3.dylib not found!

about libgfortran is somehow related to the error reported here

Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Warning in system(paste(act.cmd, collapse = " "), intern = TRUE) :
  running command '. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate && /Library/Frameworks/R.framework/Resources/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11011, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 1
Warning in socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket
Quitting from lines 90-102 (snifter.Rmd) 
Error: processing vignette 'snifter.Rmd' failed with diagnostics:
cannot open the connection

?

If you mean that the SPB needs to be updated, I guess that's what I'm looking for, the specifics of what needs to be updated for what I think is a basilisk requirement?

LTLA commented 4 years ago

I don't know what the Windows problem is - judging from the error messages, it didn't even get a chance to start the build, and that's way before any conda-related things (which only start to happen in the vignette compilation).

The Mac error is more concerning. I've never seen it before, and the Mac builds work fine on the BBS - which, I assume, uses the same settings as the SPB. It would be a stretch to assume that the Fortran linking errors are related but it's the only lead I have. One experiment we could do is:

Does /Users/pkgbuild/Library/Caches/basilisk/1.1.9/snifter-0.99.0 exist?
Does running /Users/pkgbuild/Library/Caches/basilisk/1.1.9/snifter-0.99.0/python and then import openTSNE work?
What happens with . '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate?

My suspicion is that the conda environment has been provisioned correctly but something on the SPB is interfering with correct activation of the environment, leading to the errors observed above. I could add some defensive measures to "keep trying" if the activation fails, but activation does seem kind of important so we should try to figure out why 3 doesn't work.

Note that this is unlikely to be anything snifter-specific, we have the same issue at #1549.

hpages commented 4 years ago

My 2 cents: I don't know what snifter's problem on Mac is but I don't think it's related to bluster's BUILD BIN failure on Mac. The latter is due to the SPB using a stale version of the build_universal.sh script that we use on BBS (daily builds) to generate Mac binaries that are distributable. However snifter fails during the BUILD stage which is just running R CMD build so doesn't involve build_universal.sh.

FWIW I just noticed that the reports generated by the SPB contain things like this:

Package: Package object (1146)
Version: 0.99.0
RVersion: 4.0
BiocVersion: 3.12
...

"Package object (1146)"? Not sure when we started to see this...

LTLA commented 4 years ago

Alright. After racking my brain, my main hypothesis is a problem with activation, as alluded above.

Background

conda needs to activate its environments before we can use their contents. Okay, technically that's not true, we can use the Python libraries without activation, but that is not the recommended way because the activation process sets up environment variables that might be important to downstream packages. So ideally we'd activate.

This is achieved with the following line of code in the shell:

. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate

(The bigger picture is more complex. The above line is run in a system() subshell by basilisk.utils to recover alterations to the environment variables, which are then manually applied by Sys.setenv(). This is because I couldn't figure out how to get a system command to change the environment variables in the running R process.)

Diagnosis

The error message is somewhat informative:

Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Warning in system(paste(act.cmd, collapse = " "), intern = TRUE) :
  running command '. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate && /Library/Frameworks/R.framework/Resources/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11011, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 1
Warning in socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket
Quitting from lines 90-102 (snifter.Rmd) 
Error: processing vignette 'snifter.Rmd' failed with diagnostics:
cannot open the connection

The from conda.cli import main line is most likely generated from the actual conda command used in conda activate. It's literally in the conda executable if you do which conda after the .../etc/profile.d/conda.sh call:

#!/home/luna/.cache/basilisk/1.1.8/0/bin/python
# -*- coding: utf-8 -*-
import sys
# Before any more imports, leave cwd out of sys.path for internal 'conda shell.*' commands.
# see https://github.com/conda/conda/issues/6549
if len(sys.argv) > 1 and sys.argv[1].startswith('shell.') and sys.path and sys.path[0] == '':
    # The standard first entry in sys.path is an empty string,
    # and os.path.abspath('') expands to os.getcwd().
    del sys.path[0]

if __name__ == '__main__':
    from conda.cli import main
    sys.exit(main())

This is interesting because it indicates that the conda.sh script did run, and that the conda executable was found on the path, but the underlying Python environment failed to find the conda modules. That should not be possible because the conda executable sets the #! to point to the conda-packaged Python installation!

One possible explanation is that the SPB sets a PYTHONPATH that interferes with correct discovery of the built-in conda-provided packages. However, basilisk.utils will actually unset the PYTHONPATH beforehand:

https://github.com/LTLA/basilisk.utils/blob/e87edf5f9c3ff20f5f89cec499f0eda92c46e0e9/R/activateEnvironment.R#L45

So at this point I'm like ¯\_(ツ)_/¯, and I'm only left with conspiracy theories like (i) some kind of PYTHONHOME setting, (ii) incorrect handling of the PYTHONPATH variable by Sys.unsetenv or system on Macs, or (iii) other.

Requested diagnostic actions

1. The biggest question is - what happens if we run this outside of R?

. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate

2. What are the environment variables on the SPB in the course of an R package build?

# Raw:
export

# After running:
. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh'
export

3. Is there already a system-wide conda command on the SPB?

4. More generally, what are the differences between the SPB and BBS environments? BiocSklearn has no problems on Mac on the BBS. (It has problems on Windows, but that seems to be something else... sigh.)

hpages commented 4 years ago

Action 1 seems to work fine on merida1:

merida1:~ pkgbuild$ . '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate
(base) merida1:~ pkgbuild$

I'm doing this from the pkgbuild account which is the account from which the SPB runs its builds. However, my understanding is that things run in a very different context when run by the SPB. One thing for example is that the SPB is implemented in Python, like BBS, but, unlike the latter, the former runs in a Python virtual environment. Don't know what happens when trying to activate a conda virtual environment from a Python virtual environment. Is that even supported? @lshep can provide more details.

Now I'm wondering: If the nested virtual environment thing is the problem here, how is it that we didn't catch it during basilisk submission process?

LTLA commented 4 years ago

basilisk didn't do any activation of conda environments during its submission, so the offending lines of code were never run. I added activation because it was the correct thing to do when working with conda. I guess I could disable it, but it's odd that it fails on Mac yet succeeds on Linux; and if conda can't even find itself, what hope does it have of finding anything else?

LTLA commented 4 years ago

I would be curious to know whether a quick R CMD build and check on the BBS or even under a non-SPB pkgbuild login would pass for snifter. Then we could isolate it to SPB's specific build environment.

hpages commented 4 years ago

Everything seems to be working fine from the command line on merida1:

merida1:sandbox biocbuild$ git clone https://github.com/Alanocallaghan/snifter
Cloning into 'snifter'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 206 (delta 93), reused 166 (delta 53), pack-reused 0
Receiving objects: 100% (206/206), 367.78 KiB | 7.21 MiB/s, done.
Resolving deltas: 100% (93/93), done.

merida1:sandbox biocbuild$ R CMD build snifter
* checking for file ‘snifter/DESCRIPTION’ ... OK
* preparing ‘snifter’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building ‘snifter_0.99.0.tar.gz’

merida1:sandbox biocbuild$ R CMD check snifter_0.99.0.tar.gz
* using log directory ‘/Users/biocbuild/sandbox/snifter.Rcheck’
* using R version 4.0.2 (2020-06-22)
* using platform: x86_64-apple-darwin17.0 (64-bit)
* using session charset: UTF-8
* checking for file ‘snifter/DESCRIPTION’ ... OK
* checking extension type ... Package
* this is package ‘snifter’ version ‘0.99.0’
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking for hidden files and directories ... NOTE
Found the following hidden files and directories:
  .BBSoptions
These were most likely included in error. See section ‘Package
structure’ in the ‘Writing R Extensions’ manual.
* checking for portable file names ... OK
* checking for sufficient/correct file permissions ... OK
* checking whether package ‘snifter’ can be installed ... NOTE
Found the following notes/warnings:
  Non-staged installation was used
See ‘/Users/biocbuild/sandbox/snifter.Rcheck/00install.out’ for details.
* checking installed package size ... OK
* checking package directory ... OK
* checking ‘build’ directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking line endings in shell scripts ... OK
* checking installed files from ‘inst/doc’ ... OK
* checking files in ‘vignettes’ ... OK
* checking examples ... OK
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in ‘inst/doc’ ... OK
* checking running R code from vignettes ...
  ‘snifter.Rmd’ using ‘UTF-8’... OK
 NONE
* checking re-building of vignette outputs ... OK
* checking PDF version of manual ... OK
* DONE

Status: 2 NOTEs
See
  ‘/Users/biocbuild/sandbox/snifter.Rcheck/00check.log’
for details.

LTLA commented 4 years ago

Excellent. Okay, so it's something to do with the SPB. I bet the activation of the virtual environment is setting some kind of environment variable that is causing the conda python to look in the wrong place... but not on Linux.

I could try to reproduce this locally if someone gives me the virtualenv recipe being used by the workers. It should be a simple case of just running subprocess.call() with an R CMD build string and seeing what happens.

Or we could just write it off as an idiosyncrasy of the SPB set-up that is unlikely to happen in the wild.

lshep commented 4 years ago

the virtualenv on the SPB is set up with the virtualenv -p /usr/bin/python3 env

LTLA commented 4 years ago

No juice, I run through fine with:

# In the shell:
virtualenv -p /usr/local/bin/python3 env
source env/bin/activate
python

Followed by:

# In Python:
subprocess.call(["R", "CMD", "build", "package"])

(I'm on High Sierra.)

This is probably unsurprising given that Linux doesn't have any problems either. Do the commands above work on the SPB?

lshep commented 4 years ago

When I run the above it fails with the same as on the report.

>>> import subprocess
>>> subprocess.call(["R", "CMD", "build", "snifter"])
* checking for file ‘snifter/DESCRIPTION’ ... OK
* preparing ‘snifter’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘snifter.Rmd’ using rmarkdown
Loading required package: SingleCellExperiment
Loading required package: SummarizedExperiment
Loading required package: GenomicRanges
Loading required package: stats4
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: 'BiocGenerics'

The following objects are masked from 'package:parallel':

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from 'package:stats':

    IQR, mad, sd, var, xtabs

The following objects are masked from 'package:base':

    Filter, Find, Map, Position, Reduce, anyDuplicated, append,
    as.data.frame, basename, cbind, colnames, dirname, do.call,
    duplicated, eval, evalq, get, grep, grepl, intersect, is.unsorted,
    lapply, mapply, match, mget, order, paste, pmax, pmax.int, pmin,
    pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table,
    tapply, union, unique, unsplit, which.max, which.min

Loading required package: S4Vectors

Attaching package: 'S4Vectors'

The following object is masked from 'package:base':

    expand.grid

Loading required package: IRanges
Loading required package: GenomeInfoDb
Loading required package: Biobase
Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: DelayedArray
Loading required package: Matrix

Attaching package: 'Matrix'

The following object is masked from 'package:S4Vectors':

    expand

Loading required package: matrixStats

Attaching package: 'matrixStats'

The following objects are masked from 'package:Biobase':

    anyMissing, rowMedians

Attaching package: 'DelayedArray'

The following objects are masked from 'package:matrixStats':

    colMaxs, colMins, colRanges, rowMaxs, rowMins, rowRanges

The following objects are masked from 'package:base':

    aperm, apply, rowsum

Loading required package: ggplot2
using temporary cache /var/folders/7y/c__3_48d5y18p1_rhfs_4x9c0000gt/T//RtmpJNagTd/BiocFileCache
snapshotDate(): 2020-07-10
see ?scRNAseq and browseVignettes('scRNAseq') for documentation
downloading 1 resources
retrieving 1 resource
loading from cache
see ?scRNAseq and browseVignettes('scRNAseq') for documentation
downloading 1 resources
retrieving 1 resource
loading from cache
see ?scRNAseq and browseVignettes('scRNAseq') for documentation
downloading 1 resources
retrieving 1 resource
loading from cache
snapshotDate(): 2020-07-10
see ?scRNAseq and browseVignettes('scRNAseq') for documentation
downloading 1 resources
retrieving 1 resource
loading from cache
Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Warning in system(paste(act.cmd, collapse = " "), intern = TRUE) :
  running command '. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/snifter-0.99.0/fitsne' && /Library/Frameworks/R.framework/Resources/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11179, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 1
Warning in socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket
Quitting from lines 90-102 (snifter.Rmd) 
Error: processing vignette 'snifter.Rmd' failed with diagnostics:
cannot open the connection
--- failed re-building ‘snifter.Rmd’

SUMMARY: processing the following file failed:
  ‘snifter.Rmd’

Error: Vignette re-building failed.
Execution halted
1

LTLA commented 4 years ago

Interesting. Very interesting.

Question 1:

Can I have a look at the environment variables that are visible to R? With:

# After activating the virtual environment...
subprocess.call(["R", "--quiet", "-e", "Sys.getenv()"])

I get:

> Sys.getenv()
__CF_USER_TEXT_ENCODING
                        0x1F6:0x0:0x0
Apple_PubSub_Socket_Render
                        /private/tmp/com.apple.launchd.IEkkjHT5gE/Render
CLICOLOR                1
COLORFGBG               7;0
COLORTERM               truecolor
COMMAND_MODE            unix2003
DISPLAY                 /private/tmp/com.apple.launchd.KjmeZDbhRt/org.macosforge.xquartz:0
DYLD_FALLBACK_LIBRARY_PATH
                        /Users/luna/Software/R/release/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/server
EDITOR                  vi
HOME                    /Users/luna
ITERM_PROFILE           Default
ITERM_SESSION_ID        w0t0p0:EECF7B24-0E54-4DF5-959F-409A6BB2858D
LANG                    en_US.UTF-8
LC_TERMINAL             iTerm2
LC_TERMINAL_VERSION     3.3.12
LN_S                    ln -s
LOGNAME                 luna
LSCOLORS                ExFxBxDxCxegedabagacad
MAKE                    make
PAGER                   /usr/bin/less
PATH                    /Users/luna/Programming/bioconductor/snifter/testing/env/bin:/Users/luna/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin
PWD                     /Users/luna/Programming/bioconductor/snifter/testing
R_ARCH
R_BROWSER               /usr/bin/open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /Users/luna/Software/R/release/doc
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /Users/luna/Software/R/release
R_INCLUDE_DIR           /Users/luna/Software/R/release/include
R_LIBS_SITE
R_LIBS_USER             ~/Library/R/4.0/library
R_PAPERSIZE             a4
R_PDFVIEWER             /usr/bin/open
R_PLATFORM              x86_64-apple-darwin17.7.0
R_PRINTCMD              lpr
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /var/folders/db/v59jzn_52y39l1w_hrmywbvw0000gp/T//Rtmpf4fQA5
R_SHARE_DIR             /Users/luna/Software/R/release/share
R_STRIP_SHARED_LIB      strip -x
R_STRIP_STATIC_LIB      strip -S
R_SYSTEM_ABI            osx,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD           /usr/local/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
RSTUDIO_WHICH_R         /Users/luna/Software/R/devel/bin/R
SECURITYSESSIONID       186a8
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   1
SSH_AUTH_SOCK           /private/tmp/com.apple.launchd.PKlDAAmk2V/Listeners
TAR                     /usr/bin/tar
TERM                    xterm-256color
TERM_PROGRAM            iTerm.app
TERM_PROGRAM_VERSION    3.3.12
TERM_SESSION_ID         w0t0p0:EECF7B24-0E54-4DF5-959F-409A6BB2858D
TMPDIR                  /var/folders/db/v59jzn_52y39l1w_hrmywbvw0000gp/T/
USER                    luna
VIRTUAL_ENV             /Users/luna/Programming/bioconductor/snifter/testing/env
XPC_FLAGS               0x0
XPC_SERVICE_NAME        0
>
>
0

Question 2:

Let's try something a bit more ambitious and have a look at the environment variables during conda activation:

# After activating the virtual environment...
subprocess.call(["R", "--quiet", "-e", "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && R --quiet -e \\\"Sys.getenv()\\\"\")"])

I get a faceful of:

__CF_USER_TEXT_ENCODING
                        0x1F6:0x0:0x0
_CE_CONDA
_CE_M
Apple_PubSub_Socket_Render
                        /private/tmp/com.apple.launchd.IEkkjHT5gE/Render
CLICOLOR                1
COLORFGBG               7;0
COLORTERM               truecolor
COMMAND_MODE            unix2003
CONDA_EXE               /Users/luna/Library/Caches/basilisk/1.1.9/0/bin/conda
CONDA_PYTHON_EXE        /Users/luna/Library/Caches/basilisk/1.1.9/0/bin/python
CONDA_SHLVL             0
DISPLAY                 /private/tmp/com.apple.launchd.KjmeZDbhRt/org.macosforge.xquartz:0
DYLD_FALLBACK_LIBRARY_PATH
                        /Users/luna/Software/R/release/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/server
EDITOR                  vi
HOME                    /Users/luna
ITERM_PROFILE           Default
ITERM_SESSION_ID        w0t0p0:EECF7B24-0E54-4DF5-959F-409A6BB2858D
LANG                    en_US.UTF-8
LC_TERMINAL             iTerm2
LC_TERMINAL_VERSION     3.3.12
LN_S                    ln -s
LOGNAME                 luna
LSCOLORS                ExFxBxDxCxegedabagacad
MAKE                    make
PAGER                   /usr/bin/less
PATH                    /Users/luna/Library/Caches/basilisk/1.1.9/0/condabin:/Users/luna/Programming/bioconductor/snifter/testing/env/bin:/Users/luna/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin
PWD                     /Users/luna/Programming/bioconductor/snifter/testing
R_ARCH
R_BROWSER               /usr/bin/open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /Users/luna/Software/R/release/doc
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /Users/luna/Software/R/release
R_INCLUDE_DIR           /Users/luna/Software/R/release/include
R_LIBS_SITE
R_LIBS_USER             ~/Library/R/4.0/library
R_PAPERSIZE             a4
R_PAPERSIZE_USER        a4
R_PDFVIEWER             /usr/bin/open
R_PLATFORM              x86_64-apple-darwin17.7.0
R_PRINTCMD              lpr
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /var/folders/db/v59jzn_52y39l1w_hrmywbvw0000gp/T//Rtmp69QY04
R_SHARE_DIR             /Users/luna/Software/R/release/share
R_STRIP_SHARED_LIB      strip -x
R_STRIP_STATIC_LIB      strip -S
R_SYSTEM_ABI            osx,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD           /usr/local/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
RSTUDIO_WHICH_R         /Users/luna/Software/R/devel/bin/R
SECURITYSESSIONID       186a8
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   2
SSH_AUTH_SOCK           /private/tmp/com.apple.launchd.PKlDAAmk2V/Listeners
TAR                     /usr/bin/tar
TERM                    xterm-256color
TERM_PROGRAM            iTerm.app
TERM_PROGRAM_VERSION    3.3.12
TERM_SESSION_ID         w0t0p0:EECF7B24-0E54-4DF5-959F-409A6BB2858D
TMPDIR                  /var/folders/db/v59jzn_52y39l1w_hrmywbvw0000gp/T/
USER                    luna
VIRTUAL_ENV             /Users/luna/Programming/bioconductor/snifter/testing/env
XPC_FLAGS               0x0
XPC_SERVICE_NAME        0

lshep commented 4 years ago

Question 1 results:

>>> import subprocess
>>> subprocess.call(["R", "--quiet", "-e", "Sys.getenv()"])
> Sys.getenv()
__CF_USER_TEXT_ENCODING
                        0x1FA:0:0
__PYVENV_LAUNCHER__     /Users/pkgbuild/packagebuilder/env/bin/python
COLUMNS                 80
DISPLAY                 :1.0
DYLD_FALLBACK_LIBRARY_PATH
                        /Library/Frameworks/R.framework/Resources/lib:/usr/local/jdk-14.0.1.jdk/Contents/Home/lib/server
EDITOR                  vi
HOME                    /Users/pkgbuild
JAVA_HOME               /usr/local/jdk-14.0.1.jdk/Contents/Home
LANG                    en_US.UTF-8
LINES                   24
LN_S                    ln -s
LOGNAME                 pkgbuild
MAIL                    /var/mail/pkgbuild
MAKE                    make
OPENSSL_LIBS            /usr/local/Cellar/openssl@1.1/1.1.1g/lib/libssl.a
                        /usr/local/Cellar/openssl@1.1/1.1.1g/lib/libcrypto.a
PAGER                   /usr/bin/less
PATH                    /Users/pkgbuild/packagebuilder/env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/ensembl-vep
PKG_CONFIG_PATH         /usr/local/opt/openssl@1.1/lib/pkgconfig
PWD                     /Users/pkgbuild/packagebuilder/Temp
R_ARCH                  
R_BROWSER               /usr/bin/open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /Library/Frameworks/R.framework/Resources/doc
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /Library/Frameworks/R.framework/Resources
R_INCLUDE_DIR           /Library/Frameworks/R.framework/Resources/include
R_LIBS_SITE             
R_LIBS_USER             ~/Library/R/4.0/library
R_PAPERSIZE             a4
R_PDFVIEWER             /usr/bin/open
R_PLATFORM              x86_64-apple-darwin17.0
R_PRINTCMD              lpr
R_QPDF                  /Library/Frameworks/R.framework/Resources/bin/qpdf
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /var/folders/7y/c__3_48d5y18p1_rhfs_4x9c0000gt/T//Rtmp47W9A4
R_SHARE_DIR             /Library/Frameworks/R.framework/Resources/share
R_STRIP_SHARED_LIB      strip -x
R_STRIP_STATIC_LIB      strip -S
R_SYSTEM_ABI            osx,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD           /usr/local/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   1
SSH_AUTH_SOCK           /tmp/ssh-NZmh9JznKk/agent.46810
SSH_CLIENT              67.99.175.226 19510 22
SSH_CONNECTION          67.99.175.226 19510 208.78.106.115 22
SSH_TTY                 /dev/ttys000
TAR                     /usr/bin/tar
TERM                    xterm-256color
TMPDIR                  /var/folders/7y/c__3_48d5y18p1_rhfs_4x9c0000gt/T/
USER                    pkgbuild
VIRTUAL_ENV             /Users/pkgbuild/packagebuilder/env
>

lshep commented 4 years ago

Question 2 results:

>>> subprocess.call(["R", "--quiet", "-e", "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && R --quiet -e \\\"Sys.getenv()\\\"\")"])
> system(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && R --quiet -e \"Sys.getenv()\"")
> Sys.getenv()
__CF_USER_TEXT_ENCODING
                        0x1FA:0:0
__PYVENV_LAUNCHER__     /Users/pkgbuild/packagebuilder/env/bin/python
_CE_CONDA               
_CE_M                   
COLUMNS                 80
CONDA_EXE               /Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda
CONDA_PYTHON_EXE        /Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/python
CONDA_SHLVL             0
DISPLAY                 :1.0
DYLD_FALLBACK_LIBRARY_PATH
                        /Library/Frameworks/R.framework/Resources/lib:/usr/local/jdk-14.0.1.jdk/Contents/Home/lib/server
EDITOR                  vi
HOME                    /Users/pkgbuild
JAVA_HOME               /usr/local/jdk-14.0.1.jdk/Contents/Home
LANG                    en_US.UTF-8
LINES                   24
LN_S                    ln -s
LOGNAME                 pkgbuild
MAIL                    /var/mail/pkgbuild
MAKE                    make
OPENSSL_LIBS            /usr/local/Cellar/openssl@1.1/1.1.1g/lib/libssl.a
                        /usr/local/Cellar/openssl@1.1/1.1.1g/lib/libcrypto.a
PAGER                   /usr/bin/less
PATH                    /Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/condabin:/Users/pkgbuild/packagebuilder/env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/usr/local/opt/openssl@1.1/bin:/usr/local/ensembl-vep
PKG_CONFIG_PATH         /usr/local/opt/openssl@1.1/lib/pkgconfig
PWD                     /Users/pkgbuild/packagebuilder/Temp
R_ARCH                  
R_BROWSER               /usr/bin/open
R_BZIPCMD               /usr/bin/bzip2
R_DOC_DIR               /Library/Frameworks/R.framework/Resources/doc
R_GZIPCMD               /usr/bin/gzip
R_HOME                  /Library/Frameworks/R.framework/Resources
R_INCLUDE_DIR           /Library/Frameworks/R.framework/Resources/include
R_LIBS_SITE             
R_LIBS_USER             ~/Library/R/4.0/library
R_PAPERSIZE             a4
R_PAPERSIZE_USER        a4
R_PDFVIEWER             /usr/bin/open
R_PLATFORM              x86_64-apple-darwin17.0
R_PRINTCMD              lpr
R_QPDF                  /Library/Frameworks/R.framework/Resources/bin/qpdf
R_RD4PDF                times,inconsolata,hyper
R_SESSION_TMPDIR        /var/folders/7y/c__3_48d5y18p1_rhfs_4x9c0000gt/T//RtmpMgx9kr
R_SHARE_DIR             /Library/Frameworks/R.framework/Resources/share
R_STRIP_SHARED_LIB      strip -x
R_STRIP_STATIC_LIB      strip -S
R_SYSTEM_ABI            osx,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD           /usr/local/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
SED                     /usr/bin/sed
SHELL                   /bin/bash
SHLVL                   2
SSH_AUTH_SOCK           /tmp/ssh-NZmh9JznKk/agent.46810
SSH_CLIENT              67.99.175.226 19510 22
SSH_CONNECTION          67.99.175.226 19510 208.78.106.115 22
SSH_TTY                 /dev/ttys000
TAR                     /usr/bin/tar
TERM                    xterm-256color
TMPDIR                  /var/folders/7y/c__3_48d5y18p1_rhfs_4x9c0000gt/T/
USER                    pkgbuild
VIRTUAL_ENV             /Users/pkgbuild/packagebuilder/env

LTLA commented 4 years ago

And just to be sure, this fails for the SPB:

# After activating the virtual environment...
subprocess.call(["R", "--quiet", "-e", "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate\")"])

lshep commented 4 years ago

yep

>>> import subprocess
>>> subprocess.call(["R", "--quiet", "-e", "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate\")"])
> system(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate")
Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

LTLA commented 4 years ago

Alright, just one more thing before I go and try to digest all of this. What happens if you activate the environment and run:

# After activating the virtual environment:
. '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate

lshep commented 4 years ago

merida1:packagebuilder pkgbuild$ source env/bin/activate
(env) merida1:packagebuilder pkgbuild$ python 
Python 3.7.7 (default, Mar 10 2020, 15:43:03) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(env) merida1:packagebuilder pkgbuild$ . '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate
(base) (env) merida1:packagebuilder pkgbuild$

LTLA commented 4 years ago

Sorry, actually one more thing. Is this also successful after activating the virtual env?

# After activating the virtual env...
R --quiet -e "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate\")"

lshep commented 4 years ago

merida1:packagebuilder pkgbuild$ source env/bin/activate
(env) merida1:packagebuilder pkgbuild$ R --quiet -e "system(\". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate\")"
> system(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate")
>

LTLA commented 4 years ago

Well. The only other thing I can think of is whether any of this fails:

# Inside a virtual environment again...
subprocess.call(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate", shell=True)
subprocess.call("R --quiet -e 'system(\". \'/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh\' && conda activate\")'", shell=True)

It's something to do with operating inside subprocess, given that working inside the virtual environment does not directly interfere with activation. The remaining question is whether it's a problem of subprocess itself or whether there's some kind of toxic interaction with R. Possible also for the default shell=False to be relevant here...

lshep commented 4 years ago

merida1:packagebuilder pkgbuild$ source env/bin/activate
(env) merida1:packagebuilder pkgbuild$ python3
Python 3.7.7 (default, Mar 10 2020, 15:43:03) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.call(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate", shell=True)
Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
1
>>> subprocess.call("R --quiet -e 'system(\". \'/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh\' && conda activate\")'", shell=True)
> system(". /Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh && conda activate")
Traceback (most recent call last):
  File "/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

LTLA commented 4 years ago

Well, I'm fast approaching the "stumped" state.

One more go. Can you modify /Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/bin/conda to have, on line 13:

#!/Users/luna/Library/Caches/basilisk/1.1.9/0/bin/python
# -*- coding: utf-8 -*-
import sys

# Before any more imports, leave cwd out of sys.path for internal 'conda shell.*' commands.
# see https://github.com/conda/conda/issues/6549
if len(sys.argv) > 1 and sys.argv[1].startswith('shell.') and sys.path and sys.path[0] == '':
    # The standard first entry in sys.path is an empty string,
    # and os.path.abspath('') expands to os.getcwd().
    del sys.path[0]

if __name__ == '__main__':
    print(sys.path, file=sys.stderr) ##### <---- add here (AARON)
    from conda.cli import main
    sys.exit(main())

And then run, with virtualenv activation:

subprocess.call(". '/Users/pkgbuild/Library/Caches/basilisk/1.1.9/0/etc/profile.d/conda.sh' && conda activate", shell=True)

This gives me:

['/Users/luna/Library/Caches/basilisk/1.1.9/0/bin', '/Users/luna/Library/Caches/basilisk/1.1.9/0/lib/python37.zip', '/Users/luna/Library/Caches/basilisk/1.1.9/0/lib/python3.7', '/Users/luna/Library/Caches/basilisk/1.1.9/0/lib/python3.7/lib-dynload', '/Users/luna/Library/Caches/basilisk/1.1.9/0/lib/python3.7/site-packages']
0

For which conda exists in /Users/luna/Library/Caches/basilisk/1.1.9/0/lib/python3.7/site-packages.

alanocallaghan commented 4 years ago

I hope it's obvious at this point that I reached the "stumped" state much more rapidly than Aaron. The only worthwhile thing I can think of doing is trying to identify any differences between this and BiocSklearn, which I will try tomorrow, though if it's a BBS vs SPB difference that may be futile.

LTLA commented 4 years ago

Yes, if @lshep can try the last thing I suggested, it might give some more insight. It probably won't, but who knows.

I would propose that, if snifter passes CHECK on the BBS (perhaps with some help from @hpages?), we just ignore the Mac failure on the SPB for the time being. This might apply to all basilisk clients: I recall reading that conda and virtual environments should not be run together, so it might just be good luck that everything still works on linux and on my local Mac.

hpages commented 4 years ago

Sounds good to me.

LTLA commented 4 years ago

Any news?

lshep commented 4 years ago

Sorry I was out on vacation. I'll review the package assuming its some quirk with the virtualenv that the SPB uses. I should have a review in a few days.

lshep commented 4 years ago

So I'm trying to build and install on my local machine, but I get the following when R CMD build?

Quitting from lines 90-102 (snifter.Rmd) 
Error: processing vignette 'snifter.Rmd' failed with diagnostics:
ImportError: /home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/_lib/_uarray/_uarray.cpython-37m-x86_64-linux-gnu.so: symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_createERmm, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference

Detailed traceback: 
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/openTSNE/__init__.py", line 1, in <module>
    from .tsne import TSNE, TSNEEmbedding, PartialTSNEEmbedding, OptimizationInterrupt
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/openTSNE/tsne.py", line 9, in <module>
    from sklearn.base import BaseEstimator
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/sklearn/__init__.py", line 80, in <module>
    from .base import clone
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/sklearn/base.py", line 21, in <module>
    from .utils import _IS_32BIT
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 20, in <module>
    from scipy.sparse import issparse
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/__init__.py", line 155, in <module>
    from . import fft
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/fft/__init__.py", line 74, in <module>
    from ._basic import (
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/fft/_basic.py", line 1, in <module>
    from scipy._lib.uarray import generate_multimethod, Dispatchable
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/_lib/uarray.py", line 28, in <module>
    from ._uarray import *
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/_lib/_uarray/__init__.py", line 115, in <module>
    from ._backend import *
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/_lib/_uarray/_backend.py", line 4, in <module>
    from . import _uarray  # type: ignore

--- failed re-building ‘snifter.Rmd’

SUMMARY: processing the following file failed:
  ‘snifter.Rmd’

Error: Vignette re-building failed.
Execution halted

Advice?

alanocallaghan commented 4 years ago

I'll have a look

LTLA commented 4 years ago

I assume you're on Ubuntu? I have no issues with R CMD build on 18.04.

I wonder whether you could try a few things:

Is it a conda issue?

On the command line:

/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/bin/python

and then:

import openTSNE

Is it a reticulate issue?

library(reticulate)
use_condaenv("/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/", required=TRUE)
X <- import("openTSNE")

Is it a basilisk issue?

library(basilisk)
useBasiliskEnv("/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne")
X <- import("openTSNE")

lshep commented 4 years ago

Seems like reticulate issue

> library(reticulate)
> use_condaenv("/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/", required=TRUE)
> X <- import("openTSNE")
Error in py_module_import(module, convert = convert) : 
  ImportError: /home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/scipy/_lib/_uarray/_uarray.cpython-37m-x86_64-linux-gnu.so: symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_createERmm, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference

Detailed traceback: 
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/openTSNE/__init__.py", line 1, in <module>
    from .tsne import TSNE, TSNEEmbedding, PartialTSNEEmbedding, OptimizationInterrupt
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/openTSNE/tsne.py", line 9, in <module>
    from sklearn.base import BaseEstimator
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/sklearn/__init__.py", line 80, in <module>
    from .base import clone
  File "/home/lori/.cache/basilisk/1.1.9/snifter-0.99.0/fitsne/lib/python3.7/site-packages/sklear

I'm going to see if I can use the docker to review since this may be my own issue. I am using Ubuntu but with a very out-dated OS version. If I can't install on the docker I may ask one of the other team members to try and install locally and if they have no issues take over the review.

lshep commented 4 years ago

I was able to run on the docker image successfully and encountered no issues. Since Herve already tested manually on merida1 I'm inclined to move forward with the assumption that it is indeed a conflict with the virtualenv and conda. I have no further comments or additions to the package. Cheers

bioc-issue-bot commented 4 years ago

Your package has been accepted. It will be added to the Bioconductor nightly builds.

Thank you for contributing to Bioconductor!

LTLA commented 4 years ago

Thanks @lshep, let's cross our fingers and hope for the best.

hpages commented 4 years ago

FWIW I get the exact same R CMD build error than Lori on my laptop (Ubuntu 16.04.7 LTS). I checked again on malbec1 (Ubuntu 18.04.5 LTS) and everything works fine there. Let's just hope that not too many users are still on Ubuntu < 18.04.

I've been planning to upgrade my laptop to Ubuntu 20.04 for a while now but (1) that means sacrificing a weekend just for that and (2) I'm scared!!

alanocallaghan commented 4 years ago

That would make sense, I'm on Ubuntu 18 and I've not had issues, and the CI is running 18 as well I think

fwiw I found the 16->18 and 18->20 upgrades fairly painless other than the unity to gnome transition

hpages commented 4 years ago

I don't trust the automatic upgrading. Chances are high that it'll leave the system in a wacky state. Already happened once to me. Plenty of scary messages during the upgrade process and then the machine got stale during the boot sequence after the first reboot. Luckily it was not my profesional laptop, only a build machine ;-)

For my pro laptop I reinstall the OS from scratch. It's the only true way. But yeah, it's a lot more work :-/

lshep commented 4 years ago

I have a new laptop on the way but my current one is 14. Is it worth adding a note in the README or DESCRIPTION system dependency to be >= 18

alanocallaghan commented 4 years ago

I tend to just do a full backup first - makes the upgrade process much less exciting :)

Good idea, I'll add a note stating a tentative dependency

mtmorgan commented 4 years ago

The master branch of your GitHub repository has been added to Bioconductor's git repository.

To use the git.bioconductor.org repository, we need an 'ssh' key to associate with your github user name. If your GitHub account already has ssh public keys (https://github.com/Alanocallaghan.keys is not empty), then no further steps are required. Otherwise, do the following:

See further instructions at

https://bioconductor.org/developers/how-to/git/

for working with this repository. See especially

https://bioconductor.org/developers/how-to/git/new-package-workflow/ https://bioconductor.org/developers/how-to/git/sync-existing-repositories/

to keep your GitHub and Bioconductor repositories in sync.

Your package will be included in the next nigthly 'devel' build (check-out from git at about 6 pm Eastern; build completion around 2pm Eastern the next day) at

https://bioconductor.org/checkResults/

(Builds sometimes fail, so ensure that the date stamps on the main landing page are consistent with the addition of your package). Once the package builds successfully, you package will be available for download in the 'Devel' version of Bioconductor using BiocManager::install("snifter"). The package 'landing page' will be created at

https://bioconductor.org/packages/snifter

If you have any questions, please contact the bioc-devel mailing list (https://stat.ethz.ch/mailman/listinfo/bioc-devel); this issue will not be monitored further.

Bioconductor / Contributions

snifter #1565

Background

Diagnosis

Requested diagnostic actions

Question 1:

Question 2:

Is it a conda issue?

Is it a reticulate issue?

Is it a basilisk issue?