basilisk - Githubissues

LTLA commented 4 years ago

Update the following URL to point to the GitHub repository of the package you wish to submit to Bioconductor

Repository: https://github.com/LTLA/basilisk

Confirm the following by editing each check box to '[x]'

[x] I understand that by submitting my package to Bioconductor, the package source and all review commentary are visible to the general public.
[x] I have read the Bioconductor Package Submission instructions. My package is consistent with the Bioconductor Package Guidelines.
[x] I understand that a minimum requirement for package acceptance is to pass R CMD check and R CMD BiocCheck with no ERROR or WARNINGS. Passing these checks does not result in automatic acceptance. The package will then undergo a formal review and recommendations for acceptance regarding other Bioconductor standards will be addressed.
[x] My package addresses statistical or bioinformatic issues related to the analysis and comprehension of high throughput genomic data.
[x] I am committed to the long-term maintenance of my package. This includes monitoring the support site for issues that users may have, subscribing to the bioc-devel mailing list to stay aware of developments in the Bioconductor community, responding promptly to requests for updates from the Core team in response to changes in R or underlying software.

I am familiar with the essential aspects of Bioconductor software management, including:

[x] The 'devel' branch for new packages and features.
[x] The stable 'release' branch, made available every six months, for bug fixes.
[x] Bioconductor version control using Git (optionally via GitHub).

For help with submitting your package, please subscribe and post questions to the bioc-devel mailing list.

bioc-issue-bot commented 4 years ago

Hi @LTLA

Thanks for submitting your package. We are taking a quick look at it and you will hear back from us soon.

The DESCRIPTION file for this package is:

Package: basilisk
Version: 0.99.0
Date: 2019-12-10
Title: Freezing Python Versions Inside Bioconductor Packages
Authors@R: c(person("Aaron", "Lun", role=c("aut", "cre", "cph"),
        email="infinite.monkeys.with.keyboards@gmail.com"))
Imports: 
    reticulate,
    parallel,
    utils,
    methods
Suggests:
    knitr,
    rmarkdown,
    BiocStyle,
    testthat,
    devtools,
    callr,
    BiocFileCache
biocViews: Infrastructure
Description: 
    Installs a self-contained Python instance that is managed by the R installation.
    This aims to provide a consistent Python version that can be used reliably by Bioconductor packages.
    Module versions are also controlled to guarantee consistent behavior on different user systems.
License: GPL-3
RoxygenNote: 7.0.2
StagedInstall: false
VignetteBuilder: knitr

bioc-issue-bot commented 4 years ago

A reviewer has been assigned to your package. Learn what to expect during the review process.

IMPORTANT: Please read the instructions for setting up a push hook on your repository, or further changes to your repository will NOT trigger a new build.

mtmorgan commented 4 years ago

Perfect 'reviewer' assignment!

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

c03d29d Corrected windows download URL.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

hpages commented 4 years ago

Perfect 'reviewer' assignment!

my punishment for suggesting this

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

5dbad88 Avoid messing up user's WORKON_HOME in examples. fefc25b Bumped version.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

LTLA commented 4 years ago

Okay, to sum up the first round of attempts:

You might want to clean up /home/pkgbuild/.virtualenvs/ on malbec, I forgot that the documentation examples were throwing up in there. This is now fixed.
Errors from Windows are as anticipated; according to @grimbough, the installer does not respect file paths unless they have the "proper" windows separators, e.g., C:\blah\blah\blah.
The error from celaya is definitely weird. It seems to be unable to find zlib. It works on my Mac, though I don't know whether that's because all the dependencies were already brewed.

hpages commented 4 years ago

You might want to clean up /home/pkgbuild/.virtualenvs/ on malbec

Hopefully Lori @lshep can help with this. She might need a little bit more details about what needs to be cleaned though.

It seems to be unable to find zlib. It works on my Mac, though I don't know whether that's because all the dependencies were already brewed.

We didn't brew zlib on the Mac builders. zlib is shipped with the OS (El Capitan) and is in the standard place:

merida1:~ biocbuild$ ls -l /usr/lib/libz.*
lrwxr-xr-x  1 root  wheel      12 Mar 16  2017 /usr/lib/libz.1.1.3.dylib -> libz.1.dylib
lrwxr-xr-x  1 root  wheel      12 Mar 16  2017 /usr/lib/libz.1.2.5.dylib -> libz.1.dylib
-rwxr-xr-x  1 root  wheel  177728 Jul  8  2016 /usr/lib/libz.1.dylib
lrwxr-xr-x  1 root  wheel      12 Mar 16  2017 /usr/lib/libz.dylib -> libz.1.dylib

Seems like there should be no need to brew a new zlib on top of this.

LTLA commented 4 years ago

She might need a little bit more details about what needs to be cleaned though.

There's just a few my_package_* dirs that were created during my previous examples (not anymore, dumped to tmpdir() now). They're not harmful, but no point having them stick around.

Seems like there should be no need to brew a new zlib on top of this.

The brew thing is just a bit of trivia on my set-up, for which the installation works. Don't know why it's not finding libz.1.dylib on the build machines... seems like it's right there.

Why is the installation prefix:

/private/var/folders/gh/37kcv9bj20v2qnpgxq14hh1c0000gq/T/Rtmpi0I86G/Rinst14dab52247fc5/basilisk/anaconda

? I would have thought there was more capital letters in a typical Mac system path, as reflected in:

/Users/pkgbuild/Library/Caches/BiocFileCache/13df86328173f_Anaconda3-2019.10-MacOSX-x86_64.sh

For example, my installation prefix is:

/Users/luna/Software/R/trunk/library/basilisk

I assume the installation is going to some temp location somewhere where there is no cross-device linkage. Running R CMD build --keep-empty-dirs --no-resave-data package/ works fine for me.

hpages commented 4 years ago

Don't know why it's not finding libz.1.dylib on the build machines

Does it for you after you remove your brewed zlib?

works fine for me

How much your system diverges from a barebone Mac OS installation? Wouldn't it make sense that you develop basilisk on a system that is as pristine as possible?

hpages commented 4 years ago

FWIW installation prefix /private/var/folders/gh/37kcv9bj20v2qnpgxq14hh1c0000gq/T/Rtmpi0I86G/Rinst14dab52247fc5/... is just the package temp installation folder. Consistent with what tempdir() gives me on celaya2:

> tempdir()
[1] "/var/folders/gh/37kcv9bj20v2qnpgxq14hh1c0000gq/T//RtmpnrlGFL"

(looks like /var is a symlink to /private/var on Mac, at least on El Capitan). So nothing outrageously shocking.

OTOH your installation prefix (/Users/luna/Software/R/trunk/library/basilisk) is surprising to me. Doesn't look like you're using the most standard/barebone/pristine R setup. Are you using Simon's binary? (http://mac.r-project.org/). That's what we use on celaya2.

LTLA commented 4 years ago

Using Simon's binary fails to recapitulate the error, though I'll admit I've still got all this brew stuff lying around that I don't really want to get rid of. Am trying to cajole a colleague into building this on a pristine no-brew mac. I wonder... does the celaya error just show up on build, or also install?

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

4e784df Brutal hack to force vignette build on BioC.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

89989f0 Switch to a different hack to see if it builds.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

LTLA commented 4 years ago

Very interesting. Looks like Anaconda is already installed on celaya, based on the error message:

===============================

 R CMD BUILD

===============================

* checking for file basilisk/DESCRIPTION ... OK
* preparing basilisk:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
      -----------------------------------
* installing *source* package basilisk ...
** using non-staged installation via StagedInstall field
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
ERROR: File or directory already exists: '/Users/pkgbuild/anaconda3'
If you want to update an existing installation, use the -u option.
Error: package or namespace load failed for basilisk:
 .onLoad failed in loadNamespace() for 'basilisk', details:
  call: system.file(.core_dir, package = "basilisk", mustWork = TRUE)
  error: no file found
Error: loading failed
Execution halted
ERROR: loading failed
* removing /private/var/folders/gh/37kcv9bj20v2qnpgxq14hh1c0000gq/T/RtmpKctpHb/Rinst109c9210bb205/basilisk
      -----------------------------------
ERROR: package installation failed

I could add -u to see if it still goes through, but I don't know whether this will break other things.

hpages commented 4 years ago

Yes, unlike BBS (daily builds) which runs under the biocbuild account, the SPB (Single Package Builder) runs under the pkgbuild user. The SPB is also implemented in Python but, unlike BBS, it is run in Anaconda and uses virtual envs.

I must be missing something but I thought the idea of basilisk was to always install its own Anaconda (under the package installation folder) so would be completely agnostic of whatever Python/Anaconda/modules are already installed on the system?

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

a9f8ca4 Unset fallback LIBPATH envar for Mac installation.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, TIMEOUT, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

LTLA commented 4 years ago

I must be missing something but I thought the idea of basilisk was to always install its own Anaconda (under the package installation folder) so would be completely agnostic of whatever Python/Anaconda/modules are already installed on the system?

That is correct, I was just trying to test what was wrong with the Anaconda installer by stripping back the arguments.

But in any case, I figured it out with the help of a colleague. It was some serious detective work; apparently, a pristine R installation will set the DYLD_FALLBACK_LIBRARY_PATH environment variable, which presumably causes any rpath search to skip the default library locations. This means that the system call to the Anaconda installer needs to unset this value before attempting to install anything.

So, now we can get through the initial build step. Interestingly, binary build is okay but CHECK is timing out; I had thought it would be the other way around. It seems the Mac is half speed compared to Linux for virtual environment setup... also, were we meant to get 40 minutes?

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

da9324e Switch to backslashes to attempt to fix Windows in...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

3213c53 Added some debugging messages to see what's happen...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

hpages commented 4 years ago

were we meant to get 40 minutes?

Oh right! 40 minutes is for the daily builds. The SPB allows only about 15 min. I forgot about this. The idea was to be stricter on submissions. After acceptance, we can loosen the rules a little bit.

Of course basilisk needs more than 15 min. I don't know if this could be loosen for basilisk e.g. via a new .BBSoptions and how hard it would be to adjust the SPB to support that. @lshep ?

basilisk might even need more than 40 min. (e.g. on Windows) so we'll probably need to do something on BBS too.

LTLA commented 4 years ago

It seems that we're now fine on celaya2; perhaps it was just a transient network delay, though we're still cutting it pretty close (12 minutes for CHECK as of the last build). It would be nice to have a higher safety margin, but the main relief is that it does in fact build.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

94b23db Test availability of alternative Python executable...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

da639fe Deliberately throw an error on windows to inspect ...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

grimbough commented 4 years ago

FYI I can take a look at the windows builds Monday, now I've got some other deadlines out of the way.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

fea1c2b Examine errors from the anaconda installer.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

ff6b7f0 More debugging by listing the files in the Library...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, TIMEOUT, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

1a4c221 List out all available executables in the anaconda...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, TIMEOUT, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

LTLA commented 4 years ago

On Windows, it seems like the installer is finishing without error but the installation itself is not complete. ContinuumIO/anaconda-issues#6258 suggests that this is due to some unknown security subsystem on Windows quarantining the conda files. This is twilight zone stuff.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

725f4b8 Subvert Windows file quarantining.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, TIMEOUT, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

099873d Fixed the URL for miniconda.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, TIMEOUT, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

724c7c1 Show the status of the installation directory on W...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR, TIMEOUT, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

c7c1cd5 Try installing an older version of Anaconda.

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

Bioconductor / Contributions

basilisk #1318