Open eddelbuettel opened 1 month ago
eddelbuettel
commented
1 month ago @coatless While you may be vacationing, a quick low-tech question: Do you still have, or can construct, oldrel-macos ? I am not quite sure how to go about reproducing this aside from bugging^Hkindly asking friendly folks like you :grinning:
coatless
commented
1 month ago @eddelbuettel I'll be back tomorrow to poke at it. I think it's possible to recreate ARM by using macos-latest or macos-14 images (these are intel-free AFAIK).
Another quick trick:
https://github.com/mxschmitt/action-tmate
(The other excellent approach by the fastai folks seems to be TKO'd: https://github.com/fastai/fastmac/)
coatless
commented
1 month ago The failing line on R 4.3.3 arm64 is:
causing:
Error in as.integer(as_r_value(x$indices)) :
cannot coerce type 'environment' to vector of type 'integer'Though, this looks to be affected by all sp$*_matrix() matrice calls.
Error in as.integer(as_r_value(x$row)) :
cannot coerce type 'environment' to vector of type 'integer' Error in as.integer(as_r_value(x$indices)) :
cannot coerce type 'environment' to vector of type 'integer'I think this might be related to the underlying numpy v2.0.0 change. So, maybe this is an upstream reticulate problem?
My reticulate environment has:
numpy 2.0.0 numpy=2.0.0 conda-forge
scipy 1.14.0 scipy=1.14.0 conda-forgeThe names on the mat NumPy 2 object are:
names(mat)
[1] "all" "any" "argmax" "argmin" "argpartition" "argsort"
[7] "astype" "base" "byteswap" "choose" "clip" "compress"
[13] "conj" "conjugate" "copy" "ctypes" "cumprod" "cumsum"
[19] "data" "device" "diagonal" "dot" "dtype" "dump"
[25] "dumps" "fill" "flags" "flat" "flatten" "getfield"
[31] "imag" "item" "itemset" "itemsize" "max" "mean"
[37] "min" "mT" "nbytes" "ndim" "newbyteorder" "nonzero"
[43] "partition" "prod" "ptp" "put" "ravel" "real"
[49] "repeat" "reshape" "resize" "round" "searchsorted" "setfield"
[55] "setflags" "shape" "size" "sort" "squeeze" "std"
[61] "strides" "sum" "swapaxes" "T" "take" "to_device"
[67] "tobytes" "tofile" "tolist" "tostring" "trace" "transpose"
[73] "var" "view" Relevant reticulate code:
coatless
commented
1 month ago @t-kalinowski any thoughts on why the py_to_r() conversion for sparse matrices fails on the R 4.3.3 macOS release? (See the above comment for details on debugging.)
t-kalinowski
commented
1 month ago Hmm, I can't reproduce the error locally. @coatless, were you able to reproduce the error? Any MRE you can share?
Do we know what Python reticulate is discovering on the macOS CRAN machine?
eddelbuettel
commented
1 month ago Yep, after @coatless made that connection I also started to wonder if the CRAN machine(s) are simple 'out'? This is a very bizarre, and very one-off issue on what is fairly old and stable code (to check on sparse matrices via scipy). We only see it here and it is clearly confined to oldrel AND macos across both hardware flavors.
t-kalinowski
commented
1 month ago It seems like reticulate also just started failing CRAN checks for oldrel macos. https://cran.rstudio.org/web/checks/check_results_reticulate.html
I'm thinking it is a spurious CRAN issue until I hear otherwise or I can reproduce the error locally. Hopefully will be resolved soon. Might be worth an email to CRAN, just to confirm our understanding.
eddelbuettel
commented
1 month ago Good thinking, should have checked there myself. Concur that 'sit and wait' seems like the right move. And thanks for the added adult oversight here, appreciate it.
coatless
commented
1 month ago @eddelbuettel @t-kalinowski I don't think it's a one-off issue as I was able to reproduce it locally on a fresh install. I know NumPy v2 just went live and there have been some issues with the underlying library changes.
# Setup rig
brew tap r-lib/rig
brew install --cask rig
# Install oldrel
rig add oldrel
# Set the default version (or use the menu)
rig default oldrelVerify with:
R --version# Clone repo
gh repo clone RcppCore/RcppArmadillo
# Switch into repo
cd RcppArmadilloOpen R in the RcppArmadillo folder:
RRun:
# Install reticulate
install.packages("reticulate")
# Load reticulate
library(reticulate)
# Install scipy (and subsequently `numpy`)
reticulate::py_install("scipy")
# View package list of environment
reticulate::py_list_packages()
# Run tiny test
tinytest::run_test_dir()if (! py_module_available("scipy")) exit_file("Module scipy missing")
np <- import("numpy")
mat <- np$array(list(list(1, 0, 4), list(0, 0, 5), list(2, 3, 6)))
sp <- import("scipy.sparse")
csc <- sp$csc_matrix(mat)
# Error in as.integer(as_r_value(x$indices)) :
# cannot coerce type 'environment' to vector of type 'integer'Thanks, I could reproduce.
It seems that install.packages() installs an older version of reticulate. Can you double check what version of reticulate you have installed? The current release is 1.38.0. However, install.packages("reticulate") on R-4.3.3 pulled down reticulate 1.37.0 for me.
coatless
commented
1 month ago @t-kalinowski using 1.38.0 compiled from source works nicely! Odd that the 1.37.0 binary being served by CRAN is bad.
eddelbuettel
commented
1 month ago Paging @s-u then for CRAN macOS maintenance: we have seen RcppArmadillo fail on r-oldrel on macos (both flavours), and @t-kalinowski notes that reticulate does the same. Per some tracing by @coatless this appears due to an outdated reticulate that is one version behind. It would be lovely if you could bring it to 1.38, the current release.
eddelbuettel
commented
1 month ago Paging @s-u again: Any chance you could update reticulate which (per this discussion) seems to be the driver of why r-oldrel-macos is failing (at least for) reticulate and RcppArmadillo ?
s-u
commented
1 month ago @eddelbuettel Sorry, but 1.38 does fail its checks: https://www.r-project.org/nosvn/R.check/r-oldrel-macos-arm64/reticulate-00check.html
so you have to get the reticulate maintainers to fix it.
eddelbuettel
commented
1 month ago Hi @s-u and thanks for reporting back. If you read a few of the comments above yours here you will see that a) the reticulate maintainer @t-kalinowski is here and b) that @coatless (who as you know also knows his way around a macOS installation) reported success with it. It would be great if you good folks could sort this out; else I will be forced to pull the rip-cord and skip the tests involving sparse matrices (via SciPy) if I see Darwin and R < 4.4.0. Which would not be the worst thing in the world, but I think looking at reticulate for r-oldrel is worth a few minutes of attention.
t-kalinowski
commented
1 month ago It seems that the macOS CRAN machine is attempting to evaluate vignette chunks even though the knitr option eval=FALSE is set at the top.
This only happens on the macOS runners: https://cran.rstudio.org/web/checks/check_results_reticulate.html
The relevant lines from the current r-oldrel-macos-arm64 reticulate check output:
checking running R code from vignettes ... [11s/7s] ERROR
Errors in running code in vignettes:
when running code in ‘calling_python.Rmd’
...
> knitr::opts_chunk$set(eval = FALSE)
> difflib <- import("difflib")
When sourcing ‘calling_python.R’:
Error: could not find function "import"
Execution halted
when running code in ‘package.Rmd’
...
+ if (!have_scipy)
+ skip("scipy not available for te ..." ... [TRUNCATED]
> test_that("Things work as expected", {
+ skip_if_no_scipy()
+ })
When sourcing ‘package.R’:
Error: could not find function "test_that"
Execution halted
when running code in ‘python_packages.Rmd’
...
> library(reticulate)
> conda_create("r-reticulate")
Warning in check_forbidden_install("Conda Environments") :
cannot install Conda Environments during R CMD check
When sourcing ‘python_packages.R’:
Error: Unable to find conda binary. Is Anaconda installed?
Execution halted
when running code in ‘python_primer.Rmd’
...
Hello Friend
> library(reticulate)
> py$a_strict_Python_function(3)
When sourcing ‘python_primer.R’:
Error: AttributeError: module '__main__' has no attribute 'a_strict_Python_function'
Run `reticulate::py_last_error()` for details.
Execution halted
when running code in ‘versions.Rmd’
...
> library(reticulate)
> scipy <- import("scipy")
> scipy$amin(c(1, 3, 5, 7))
When sourcing ‘versions.R’:
Error: AttributeError: Module 'scipy' has no attribute 'amin'
Run `reticulate::py_last_error()` for details.
Execution halted
‘arrays.Rmd’ using ‘UTF-8’... [0s/0s] OK
‘calling_python.Rmd’ using ‘UTF-8’... failed
‘package.Rmd’ using ‘UTF-8’... failed
‘python_dependencies.Rmd’ using ‘UTF-8’... [0s/0s] OK
‘python_packages.Rmd’ using ‘UTF-8’... failed
‘python_primer.Rmd’ using ‘UTF-8’... failed
‘r_markdown.Rmd’ using ‘UTF-8’... [0s/0s] OK
‘versions.Rmd’ using ‘UTF-8’... failed
checking re-building of vignette outputs ... [25s/28s] OK
checking PDF version of manual ... [5s/6s] OK
DONE
Status: 1 ERROR, 1 NOTE
using check arguments '--no-clean-on-error 'The first two lines in the check output,
> knitr::opts_chunk$set(eval = FALSE)
> difflib <- import("difflib")I believe are from: https://github.com/rstudio/reticulate/blob/740169ae2a7c943115b2a4eca3b5a0b7d9e5ef3a/vignettes/calling_python.Rmd#L13 and https://github.com/rstudio/reticulate/blob/740169ae2a7c943115b2a4eca3b5a0b7d9e5ef3a/vignettes/calling_python.Rmd#L85
The second line should not be being evaluated.
(Last commit to that vignette was almost 1 year ago).
s-u
commented
1 month ago Well, don't ask me - it has nothing to do with the build machine, you can replicate it anywhere with R 4.3.3. It could well be a regression in latest rmarkdown/knitr interacting with R 4.3.3 - I don't know, but it does cause the failure so whoever causes this in the chain needs to fix it (or give up on updates in R-oldrel?).
s-u
commented
1 month ago I can confirm that this is easy to reproduce - all you need in the calling_python.Rmd in vignettes of an empty package. It's quite fascinating, because the chunks that declare just r will not be evaluated, while the chunks with {r} will be evaluated (why mixing both in one file is beyond me, but it is interesting). The chunk global default is still eval=FALSE inside the evaluated chunk it looks like one of the other option layers must be overriding it. I have no idea about the vignette pipeline, so I'll leave it to someone more familiar to dig into that one. Also there is something in the file that triggers it, because just using a random {r} chunk after setting the default does not trigger the problem.
eddelbuettel
commented
1 month ago Thanks for checking, @s-u. It's appreciated.
t-kalinowski
commented
1 month ago Thanks @s-u. Unfortunately I'm a bit stuck on this. I'm unable to reproduce the check failure locally on my mac, nor on GHA
Any assistance you can provide would be much appreciated.
t-kalinowski
commented
1 month ago Maybe the macOS runner is attempting to purl() the vignette and evaluate it that way, as part of R CMD check?
There is an active thread on knitr where multiple other packages have recently started failing on macOS-oldrel in a similar way. https://github.com/yihui/knitr/issues/2338
There is also an old thread from a few years ago where a similar failure was observed, but it didn't persist long-enough to require a proper fix. https://github.com/yihui/knitr/issues/2103
I can now reproduce the R CMD check failure with reticulate locally if I set env var
_R_CHECK_VIGNETTES_SKIP_RUN_MAYBE_=false
See https://cloud.r-project.org/web/checks/check_results_RcppArmadillo.html which currently shows failures on both 'R-oldrel-macos' for both x86_64 and arm64.