Closed barracuda156 closed 10 months ago
How did you install the package? Be aware that Bioconductor packages should always be installed with BiocManager::install()
.
@hpages Thank you for responding.
I do not see any reason to restrict a user to a specific convenience tool; what is convenient for one may not be for someone else. The only requirement is that dependencies should be satisfied, and they are in this case. The code is also unmodified, no patches applied.
I have ran tests on macOS 14 now on aarch64 and got the same errors (so OS is not to blame):
---> Testing R-BSgenomeForge
* using log directory ‘/opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_R_R-BSgenomeForge/R-BSgenomeForge/work/BSgenomeForge/BSgenomeForge.Rcheck’
* using R version 4.3.1 (2023-06-16)
* using platform: aarch64-apple-darwin23.0.0 (64-bit)
* R was compiled by
clang version 15.0.7
GNU Fortran (MacPorts gcc12 12.3.0_3+stdlib_flag) 12.3.0
* running under: macOS Sonoma 14.0
* using session charset: UTF-8
* checking for file ‘BSgenomeForge/DESCRIPTION’ ... OK
* this is package ‘BSgenomeForge’ version ‘1.2.0’
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... NOTE
Depends: includes the non-default packages:
'BiocGenerics', 'S4Vectors', 'IRanges', 'GenomeInfoDb', 'Biostrings',
'BSgenome'
Adding so many packages to the search path is excessive and importing
selectively is preferable.
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking for sufficient/correct file permissions ... OK
* checking whether package ‘BSgenomeForge’ can be installed ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking ‘build’ directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking installed files from ‘inst/doc’ ... OK
* checking files in ‘vignettes’ ... OK
* checking examples ... OK
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ...
Running ‘testthat.R’
ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-forgeBSgenomeDataPkgFromNCBI.R:32:5'): forgeBSgenomeDataPkgFromNCBI() works on "Torque teno virus 1" ──
file.exists(path) is not TRUE
`actual`: FALSE
`expected`: TRUE
── Failure ('test-forgeBSgenomeDataPkgFromUCSC.R:28:5'): forgeBSgenomeDataPkgFromUCSC() works on "eboVir3" ──
file.exists(path) is not TRUE
`actual`: FALSE
`expected`: TRUE
[ FAIL 2 | WARN 2 | SKIP 2 | PASS 40 ]
Error: Test failures
Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in ‘inst/doc’ ... OK
* checking running R code from vignettes ...
‘UsingBSgenomeForge.Rmd’ using ‘UTF-8’... OK
NONE
This should be reproducible in Macports environment if you pick the port from here: https://github.com/macports/macports-ports/commit/8ba0055fc3a109d142e2a87bbf6d18d574484ed3 (or once it is merged, then from the base normally) – and run sudo port -v test R-BSgenomeForge
.
Hi @barracuda156 -- we don't restrict anyone to a convenience tool but we are only able to support a limited number of approaches to consistent package installation. We might be able to help more if you would report sessionInfo() result and verify that BiocManager::valid() yields TRUE.
@vjcitn This is on macOS 14 (my setup on 10.6 is identical for R
packages, but uses GCC instead of Clang):
R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin23.0.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
[R.app GUI 1.79 (8225) aarch64-apple-darwin23.0.0]
[History restored from /Users/svacchanda/.Rapp.history]
> library(BSgenomeForge)
Loading required package: BiocGenerics
Attaching package: ‘BiocGenerics’
The following objects are masked from ‘package:stats’:
IQR, mad, sd, var, xtabs
The following objects are masked from ‘package:base’:
anyDuplicated, aperm, append, as.data.frame, basename, cbind,
colnames, dirname, do.call, duplicated, eval, evalq, Filter, Find,
get, grep, grepl, intersect, is.unsorted, lapply, Map, mapply,
match, mget, order, paste, pmax, pmax.int, pmin, pmin.int,
Position, rank, rbind, Reduce, rownames, sapply, setdiff, sort,
table, tapply, union, unique, unsplit, which.max, which.min
Loading required package: S4Vectors
Loading required package: stats4
Attaching package: ‘S4Vectors’
The following object is masked from ‘package:utils’:
findMatches
The following objects are masked from ‘package:base’:
expand.grid, I, unname
Loading required package: IRanges
Loading required package: GenomeInfoDb
Loading required package: Biostrings
Loading required package: XVector
Attaching package: ‘Biostrings’
The following object is masked from ‘package:base’:
strsplit
Loading required package: BSgenome
Loading required package: GenomicRanges
Loading required package: BiocIO
Loading required package: rtracklayer
Attaching package: ‘rtracklayer’
The following object is masked from ‘package:BiocIO’:
FileForFormat
> sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin23.0.0 (64-bit)
Running under: macOS Sonoma 14.0
Matrix products: default
BLAS: /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/lib/libRblas.dylib
LAPACK: /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: Asia/Taipei
tzcode source: internal
attached base packages:
[1] stats4 stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] BSgenomeForge_1.2.0 BSgenome_1.70.0 rtracklayer_1.62.0
[4] BiocIO_1.13.0 GenomicRanges_1.55.0 Biostrings_2.71.1
[7] XVector_0.43.0 GenomeInfoDb_1.39.0 IRanges_2.37.0
[10] S4Vectors_0.41.1 BiocGenerics_0.49.0
loaded via a namespace (and not attached):
[1] crayon_1.5.2 DelayedArray_0.29.0
[3] SummarizedExperiment_1.33.0 GenomicAlignments_1.38.0
[5] rjson_0.2.21 RCurl_1.98-1.12
[7] XML_3.99-0.14 MatrixGenerics_1.15.0
[9] Biobase_2.63.0 grid_4.3.1
[11] restfulr_0.0.15 abind_1.4-5
[13] bitops_1.0-7 yaml_2.3.7
[15] compiler_4.3.1 codetools_0.2-19
[17] BiocParallel_1.37.0 lattice_0.22-5
[19] SparseArray_1.3.0 parallel_4.3.1
[21] GenomeInfoDbData_1.2.11 Matrix_1.6-1.1
[23] tools_4.3.1 matrixStats_1.0.0-9003
[25] Rsamtools_2.19.0 zlibbioc_1.49.0
[27] S4Arrays_1.3.0
> library(BiocManager)
Bioconductor version 3.18 (BiocManager 1.30.22), R 4.3.1 (2023-06-16)
> BiocManager::valid()
* sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin23.0.0 (64-bit)
Running under: macOS Sonoma 14.0
Matrix products: default
BLAS: /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/lib/libRblas.dylib
LAPACK: /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: Asia/Taipei
tzcode source: internal
attached base packages:
[1] stats4 stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] BiocManager_1.30.22 BSgenomeForge_1.2.0 BSgenome_1.70.0
[4] rtracklayer_1.62.0 BiocIO_1.13.0 GenomicRanges_1.55.0
[7] Biostrings_2.71.1 XVector_0.43.0 GenomeInfoDb_1.39.0
[10] IRanges_2.37.0 S4Vectors_0.41.1 BiocGenerics_0.49.0
loaded via a namespace (and not attached):
[1] crayon_1.5.2 DelayedArray_0.29.0
[3] SummarizedExperiment_1.33.0 GenomicAlignments_1.38.0
[5] rjson_0.2.21 RCurl_1.98-1.12
[7] XML_3.99-0.14 MatrixGenerics_1.15.0
[9] Biobase_2.63.0 grid_4.3.1
[11] restfulr_0.0.15 abind_1.4-5
[13] bitops_1.0-7 yaml_2.3.7
[15] compiler_4.3.1 codetools_0.2-19
[17] BiocParallel_1.37.0 lattice_0.22-5
[19] SparseArray_1.3.0 parallel_4.3.1
[21] GenomeInfoDbData_1.2.11 Matrix_1.6-1.1
[23] tools_4.3.1 matrixStats_1.0.0-9003
[25] Rsamtools_2.19.0 zlibbioc_1.49.0
[27] S4Arrays_1.3.0
Bioconductor version '3.18'
* 3 packages out-of-date
* 37 packages too new
create a valid installation with
BiocManager::install(c(
"AnnotationDbi", "Biobase", "BiocFileCache", "BiocGenerics", "BiocIO",
"BiocParallel", "BiocStyle", "Biostrings", "blob", "cli", "DBI",
"DelayedArray", "GenomeInfoDb", "GenomicFeatures", "GenomicRanges",
"graph", "hms", "htmlwidgets", "IRanges", "KEGGREST", "knitr",
"MatrixGenerics", "matrixStats", "numDeriv", "pillar", "ps", "R6",
"RcppEigen", "Rhtslib", "rlang", "Rsamtools", "RSQLite", "S4Arrays",
"S4Vectors", "SparseArray", "SummarizedExperiment", "tibble", "withr",
"XVector", "zlibbioc"
), update = TRUE, ask = FALSE, force = TRUE)
more details: BiocManager::valid()$too_new, BiocManager::valid()$out_of_date
Warning message:
3 packages out-of-date; 37 packages too new
I can update three packages in question now and verify the result is the same.
UPD. Well, nothing Bioconductor-related is outdated here. Perhaps these were updated very recently, so I have a slightly earlier versions on this machine:
> BiocManager::valid()$out_of_date
Package
htmlwidgets "htmlwidgets"
knitr "knitr"
withr "withr"
LibPath
htmlwidgets "/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library"
knitr "/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library"
withr "/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library"
Installed Built ReposVer
htmlwidgets "1.6.1" "4.3.1" "1.6.2"
knitr "1.44" "4.3.1" "1.45"
withr "2.5.1" "4.3.1" "2.5.2"
@vjcitn @hpages Since R-BSgenomeForge
has been just merged into Macports, it should be available to try freely with sudo port -v test R-BSgenomeForge
within half an hour or so, once Macports repos sync. Make sure to run sudo port sync
first.
What BSgenomeForge source tarball does Macports use? If it was not produced with the --keep-empty-dirs
flag then it's broken. It's a Macports problem. We don't support installation via Macports.
@hpages I do not really get your reaction; I nowhere made a claim that you code is broken, I merely reported a fact that specific tests fail on macOS. (Though if the compilation requires specific uncommon flags to be passed, it can be implemented via configure instead of hoping a user to guess that.)
Also, I nowhere demanded Macports to be pro-actively supported, but how does it hurt if your package reaches some new users?
I can't do anything about Macports distributing broken packages. And this has nothing to do with configure
. The only installation method we support is via BiocManager::install()
. If you use something else, you're on your own.
I can't do anything about Macports distributing broken packages.
@hpages You can if you wish to: just advising me how to correctly compile from source is enough, I will fix it in Macports.
This is a pure R package so there's no compilation involved. But R CMD build
's option --keep-empty-dirs
must be used when making the source tarball, as mentioned above.
@hpages Thank you, I will test it now and add in the upcoming PR to Macports.
No problem.
Also about this:
The only requirement is that dependencies should be satisfied, and they are in this case.
Not with Bioconductor packages. We have 2 Bioconductor releases per year, one in Spring and one in Fall. Another requirement is that you install the right version of a Bioconductor package for the right version of Bioconductor. Most alternative installation methods fail to do that and mix packages from different Bioconductor versions, with all kinds of bad consequences. So unless you know for sure that Macports does the right thing, I would recommend that you avoid it for installing Bioconductor on your machine.
@barracuda156 I haven't looked at macports for a long time, because both Bioconductor and CRAN have very specific approaches to distributing software packages for macs. When the documented methods are followed, it seems very straightforward to obtain consistent installations using BiocManager. It was somewhat disappointing to see https://ports.macports.org/port/R-GenomicRanges/builds/ is on rtracklayer 1.55.0, which would not be a version we'd propose using with current R-patched. We don't have a lot of bandwidth at Bioconductor, but it is clear that software redistribution through e.g., macports or conda, leads to complexities that might be reduced with more collaborative communication. I made a channel #other-distros at the community bioconductor slack so we can have a more inclusive discussion if such is of interest.
So unless you know for sure that Macports does the right thing
@hpages Since I have written the whole implementation for installing R
packages in Macports and as long as I maintain it, I am reasonably sure that either things work as intended or, whenever not, I can detect that and hopefully fix.
I have updated everything we had Bioconductor-related to the current releases, so provisionally it should work. (Indeed, a late spring update missed a few packages, my fault.)
I appreciate your help with this. I am interested to have everything working correctly.
(One of the reasons I bothered to do this was to support older macOS systems, which neither CRAN nor most upstreams do – for understandable reasons of having limited resources and often lack of motivation. In result, as of now at least, we have pretty much everything working back to macOS 10.5, including powerpc and 32-bit i386.)
@vjcitn I am not sure what R-patched
is, we use 4.3.1 at the moment, but I expect upstream to release 4.4 soon, and we gonna update to that one.
Normally I run tests for packages, whenever those are provided, so generally speaking, things should work fine.
Having said that, if you recommend pegging Bioconductor stuff to released versions without updating in between, that can be done. Technically it is easier. Why I tend to go with development versions as long as they build and pass tests is that it allows to detect bugs in the process and report those before it goes into release. But we can change the policy for Bioconductor stuff here: reliable functionality is more important.
See https://bioconductor.org/news/bioc_3_18_release/ ... this release, which will be the active user-oriented version until April 2024, uses R 4.3. R 4.4 would be used only for the developer-oriented code base and so I would not recommend using it for macports. (The "R-patched" terminology refers to the fact that the code base for R 4.3 receives fairly frequent patching and the startup banner will typically reflect this.) Can we take this discussion to slack?
we use 4.3.1 at the moment, but I expect upstream to release 4.4 soon, and we gonna update to that one
BioC 3.18 (the latest release) is based on R 4.3 and BioC 3.19 (the devel version) is based on R 4.4. But BioC 3.20 will also be based on R 4.4. The question is: what version (or versions) of Bioconductor do you distribute via Macports? And how do you make sure the user gets a version that is compatible with their version of R?
BioC 3.18 (the latest release) is based on R 4.3 and BioC 3.19 (the devel version) is based on R 4.4. But BioC 3.20 will also be based on R 4.4. The question is: what version (or versions) of Bioconductor do you distribute via Macports?
@hpages Since we do not maintain multiple versions of R
(and neither other maintainers nor myself think it is needed), it looks reasonable to have Bioconductor packages versions targeting CRAN release of R
(we build ourselves from sources, but sources for R
itself are from CRAN releases).
I got the idea, it can be followed on our side, I believe. Thank you for elaborating on this.
And how do you make sure the user gets a version that is compatible with their version of R?
We cannot force anyone, but normally end-users will update ports following what Macports has. So as long as we keep it coherent in Macports at any given time, standard installations will be good. (Whatever someone does locally cannot be controlled or guaranteed, obviously.)
@vjcitn Thank you. Yes, I can register for Slack and join there. And here we wrap up.
it looks reasonable to have Bioconductor packages versions targeting CRAN release of R
I don't know what you mean with that exactly. If you're distributing BioC 3.18, you can't target R 4.4. If you're distributing BioC 3.19, then you must target R 4.4 but then you're distributing the devel version of Bioconductor which is not aimed at end users.
The discussion should be in terms of what version of Bioconductor you want to distribute/support, and whether you want (and can) follow our 6-month release cycles.
@hpages Current at any given time CRAN-released version of R
is a constraint we have.
(I would expect any release of Bioconductor to target latest release of R
, but this is not something I have any influence on, or even qualification to make a judgement.)
So if it is desirable to have all Bioconductor packages in sync to a single version of Bioconductor – which is what I understood is the case (correct me if I am wrong), what I see as a solution is to update Bioconductor packages following updates of R
. I.e. when R
4.4 is released, we update to 3.19, and then keep all Bioconductor untouched until 3.20, then update it all at once.
Whether it also implies staying at released Bioconductor and not development version gonna depend on Bioconductor decisions to release. If release will lag behind R
, we will have to switch to devel Bioconductor for the time-being.
@barracuda156 At the time of its release, a new release of Bioconductor targets the latest release of R. So the current release (3.18, released last week) targets R 4.3. The current devel version, BioC 3.19, targets R 4.4 and will be released in Spring a few days after R 4.4 itself is released.
Another important thing is to make sure that Macports gets Bioconductor source packages from the right place. Getting them from GitHub is wrong because some Bioconductor packages are not even hosted on GitHub, and, even when they are, there's no guarantee that they're in sync with what we release or that they even work. What we release comes from the Bioconductor git server at git.bioconductor.org, and it went thru our daily builds/checks to verify interoperability within all the packages that go in a given release. Given the level of inter-dependencies within the Bioconductor ecosystem and the complexity of their interactions, there is a lot that can go wrong by grabbing packages directly from GitHub. This is why we also strongly recommend against installing directly from GitHub.
Alternatively, note that you can grab source tarballs directly from our package repositories (e.g. https://bioconductor.org/package/3.18/bioc for BioC 3.18 software packages). This ensures that:
--keep-empty-dirs
)R CMD check
during our daily buildsThis is what we actually recommend for any entity that decides to re-package/re-distribute Bioconductor packages.
@hpages Thank you. We do have an option to install sources from Bioconductor (as we do support CRAN and R-forge), I just need to switch those Bioconductor ones which are presently sourced from GitHub to use sources from Bioconductor.
By the way, I can confirm that all tests pass fine now for BSgenomeForge
. Thank you for helping.
P. S. Should we add --keep-empty-dirs
to R CMD build
for all Bioconductor packages or is it specific to BSgenomeForge
?
Should we add --keep-empty-dirs to R CMD build for all Bioconductor packages
Absolutely. This is the safe thing to do, since removing empty dirs has the potential to break packages. Unfortunately that's what R CMD build
does by default but that's another discussion for another day.
What may be going with these?