reticulate error using densviz:densmap in R 4.3.1

[semmrich]

First off a huge shoutout to the OSCA Bioconductor team - this is a super detailed repository on scRNA-Seq analysis, with a level of verbosity (and fun reading!) dwarfing comparable tutorials (seurat, monocle, scanpy) imho.

Now there is an Error when I try to compute a Density preserving UMAP as outlined in OSCA

MPA.densmap <- densmap(reducedDim(MPA.sce, "PCA"), dens_frac = 0.4, dens_lambda = 0.2)

results in

Error in py_module_import(module, convert = convert) : 
TypeError: the first argument must be callable
Run reticulate::py_last_error() for details.

Any ideas what could be reason for this error?

sessionInfo()
R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    
time zone: Europe/Berlin
tzcode source: internal
attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     
other attached packages:
 [1] snifter_1.10.0              corral_1.10.0               densvis_1.10.3              ggalt_0.4.0                 clustree_0.5.0             
 [6] ggraph_2.1.0                bluster_1.10.0              scater_1.28.0               scran_1.28.2                scuttle_1.10.2             
[11] BBmisc_1.13                 lubridate_1.9.3             forcats_1.0.0               stringr_1.5.0               dplyr_1.1.3                
[16] purrr_1.0.2                 readr_2.1.4                 tidyr_1.3.0                 tibble_3.2.1                ggplot2_3.4.3              
[21] tidyverse_2.0.0             readxl_1.4.3                SingleCellExperiment_1.22.0 SummarizedExperiment_1.30.2 Biobase_2.60.0             
[26] GenomicRanges_1.52.0        GenomeInfoDb_1.36.3         IRanges_2.34.1              S4Vectors_0.38.2            BiocGenerics_0.46.0        
[31] MatrixGenerics_1.12.3       matrixStats_1.0.0          

loaded via a namespace (and not attached):
  [1] RColorBrewer_1.1-3          rstudioapi_0.15.0           jsonlite_1.8.7              MultiAssayExperiment_1.26.0 magrittr_2.0.3             
  [6] ggbeeswarm_0.7.2            farver_2.1.1                zlibbioc_1.46.0             vctrs_0.6.3                 DelayedMatrixStats_1.22.6  
 [11] RCurl_1.98-1.12             S4Arrays_1.0.6              BiocNeighbors_1.18.0        cellranger_1.1.0            proj4_1.0-13               
 [16] KernSmooth_2.23-22          basilisk_1.12.1             plyr_1.8.8                  igraph_1.5.1                lifecycle_1.0.3            
 [21] pkgconfig_2.0.3             rsvd_1.0.5                  Matrix_1.6-2                R6_2.5.1                    GenomeInfoDbData_1.2.10    
 [26] digest_0.6.33               colorspace_2.1-1            dqrng_0.3.1                 irlba_2.3.5.1               beachmat_2.16.0            
 [31] labeling_0.4.3              filelock_1.0.2              fansi_1.0.4                 timechange_0.2.0            polyclip_1.10-6            
 [36] abind_1.4-7                 compiler_4.3.1              withr_2.5.1                 backports_1.4.1             BiocParallel_1.34.2        
 [41] viridis_0.6.4               ggforce_0.4.1               Rttf2pt1_1.3.12             maps_3.4.1                  MASS_7.3-60                
 [46] DelayedArray_0.26.7         tools_4.3.1                 vipor_0.4.5                 beeswarm_0.4.0              extrafontdb_1.0            
 [51] glue_1.6.2                  grid_4.3.1                  checkmate_2.2.0             reshape2_1.4.4              cluster_2.1.4              
 [56] generics_0.1.3              gtable_0.3.4                tzdb_0.4.0                  data.table_1.14.8           hms_1.1.3                  
 [61] BiocSingular_1.16.0         tidygraph_1.2.3             ScaledMatrix_1.8.1          metapod_1.8.0               utf8_1.2.3                 
 [66] XVector_0.40.0              ggrepel_0.9.3               pillar_1.9.0                pals_1.8                    limma_3.56.2               
 [71] tweenr_2.0.2                lattice_0.21-8              tidyselect_1.2.0            locfit_1.5-9.8              transport_0.14-6           
 [76] gridExtra_2.3               edgeR_3.42.4                graphlayouts_1.0.1          statmod_1.5.0               stringi_1.7.12             
 [81] codetools_0.2-19            extrafont_0.19              cli_3.6.1                   ash_1.0-15                  reticulate_1.32.0          
 [86] munsell_0.5.0               dichromat_2.0-1             Rcpp_1.0.11                 dir.expiry_1.8.0            mapproj_1.2.11             
 [91] png_0.1-8                   parallel_4.3.1              assertthat_0.2.1            basilisk.utils_1.12.1       sparseMatrixStats_1.12.2   
 [96] bitops_1.0-7                ggthemes_4.2.4              viridisLite_0.4.2           scales_1.2.1                crayon_1.5.2               
[101] rlang_1.1.1                 cowplot_1.1.1

[lgeistlinger]

Hi @semmrich. Thanks for the feedback. Do you also encounter the problem on the sce.zeisel dataset that is used for demonstrating the functionality in the corresponding chapter of the book (just ran it, worked fine on a mac)?

If yes, this might point to problems in your python setup. If no, this might point to missing annotations in your dataset.

Btw can you also provide:

1. What you get from printing the structure of the SCE, ie just: > MPA.sce so we get an idea of the anatomy of the dataset
2. What you get from reticulate::py_last_error() after encountering the error.

Thanks!

[semmrich]

Yes, I do get the error with the zeisel dataset! And I get it on two different mashines...

Moreover, there is an Error related to python whenever I start any R seesion:

Error in FUN(X[[i]], ...) : 
  'CreateProcess' failed to run 'C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\python.exe -c "import sys; print(sys.platform)"'
Calls: do.call -> <Anonymous> -> <Anonymous> -> vapply -> FUN

Also for the other infos I should provide:

> MPA.sce
class: SingleCellExperiment 
dim: 13 477 
metadata(0):
assays(2): counts logcounts
rownames(13): total.i dead.i ... CD62P.LK.i EPCR.LK.i
rowData names(0):
colnames(477): P1_DMSO-B2 P1_DMSO-C2 ... P8_VE-F9 P8_VE-F10
colData names(19): Plate Treatment ... Concentration RBB.i
reducedDimNames(7): PCA corral ... denSNE FDG
mainExpName: NULL
altExpNames(0):

> reticulate::py_last_error()

── Python Exception Message ─────────────────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\umap\__init__.py", line 2, in <module>
    from .umap_ import UMAP
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\umap\umap_.py", line 10, in <module>
    from scipy.optimize import curve_fit
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\scipy\optimize\__init__.py", line 401, in <module>
    from ._minimize import *
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\scipy\optimize\_minimize.py", line 27, in <module>
    from ._trustregion_constr import _minimize_trustregion_constr
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\scipy\optimize\_trustregion_constr\__init__.py", line 4, in <module>
    from .minimize_trustregion_constr import _minimize_trustregion_constr
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\scipy\optimize\_trustregion_constr\minimize_trustregion_constr.py", line 5, in <module>
    from .._constraints import (
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\scipy\optimize\_constraints.py", line 8, in <module>
    from numpy.testing import suppress_warnings
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\site-packages\numpy\testing\__init__.py", line 8, in <module>
    from unittest import TestCase
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\unittest\__init__.py", line 60, in <module>
    from .async_case import IsolatedAsyncioTestCase
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\unittest\async_case.py", line 1, in <module>
    import asyncio
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\asyncio\__init__.py", line 41, in <module>
    from .windows_events import *
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\asyncio\windows_events.py", line 20, in <module>
    from . import windows_utils
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 93, in _run_hook
    module = hook()
  File "C:\Program Files\R\R-4.3.1\library\reticulate\python\rpytools\loader.py", line 117, in _hook
    return _find_and_load(name, import_)
  File "C:\Users\FUCHSD~1\AppData\Local\R\cache\R\basilisk\112~1.1\densvis\110~1.3\densvis\lib\asyncio\windows_utils.py", line 125, in <module>
    class Popen(subprocess.Popen):
TypeError: the first argument must be callable

── R Traceback ──────────────────────────────────────────────────────────────────────────────────────────────────────────
    ▆
 1. └─densvis::densmap(...)
 2.   └─densvis::umap(..., densmap = TRUE)
 3.     └─basilisk::basiliskRun(...)
 4.       └─densvis (local) fun(...)
 5.         └─reticulate::import("umap")
 6.           └─reticulate:::py_module_import(module, convert = convert)

[alanocallaghan]

Moreover, there is an Error related to python whenever I start any R seesion:

This is strange. Do you know what is calling python at the start of your R sessions? Do you have something in your .Rprofile that might be doing this? It may be interfering with the reticulate/basilisk python later on...

I can't see anything obvious in the traceback from densvis, but it seems to me to be a general python problem rather than a densvis problem, given it's a TypeError coming from rpytools/windows_utils

[semmrich]

I could resolve the starting error by deleting a stand-along python installation from the windows app store. Just aside: I never found any .Rprofile file anywhere on my PC, ofc with 'show hidden files' - not in the installation folder, nor in temporary locations. And I couldn't find it on three labtops running RStudio just fine?!

But I still cannot run densmap(), which is annoying as it worked earlier.

Interestingly, another python-interfacing dimred from OSCA works well:

MPA.sce <- runTSNE(MPA.sce, use_fitsne = TRUE, name="FIt-SNE")

[semmrich]

Now it "magically" resolved, densmap() function runs seamless, case closed. Thank You for the helpful comments!