crisprVerse / crisprScore

On-Target and Off-Target Scoring Algorithms for CRISPR gRNAs
MIT License
13 stars 3 forks source link

.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/libpython2.7.so: undefined symbol: PyType_GetFlags #8

Closed serge2016 closed 11 months ago

serge2016 commented 1 year ago

Hello!

Thank you for this package! Could you help me, please, to fix this with new versions of Ubuntu 22.04?

R -e "library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)"

#44 0.450 R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
#44 0.450 Copyright (C) 2023 The R Foundation for Statistical Computing
#44 0.450 Platform: x86_64-pc-linux-gnu (64-bit)
...
#44 0.537 > library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)
#44 1.652 trying URL 'https://repo.anaconda.com/miniconda/Miniconda3-py310_23.5.0-3-Linux-x86_64.sh'
#44 2.086 Content type 'application/x-sh' length 95815836 bytes (91.4 MB)
#44 2.189 ==================================================
#44 6.373 downloaded 91.4 MB
#44 6.373 
#44 6.387 PREFIX=/home/rstudio/.cache/R/basilisk/1.13.3/0
#44 6.627 Unpacking payload ...
...
#44 178.5   added / updated specs:
#44 178.5     - python=2.7
#44 178.5 
#44 178.5 
#44 178.5 The following packages will be downloaded:
#44 178.5 
#44 178.5     package                    |            build
#44 178.5     ---------------------------|-----------------
#44 178.5     _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
#44 178.5     _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
#44 178.5     ca-certificates-2023.7.22  |       hbcca054_0         146 KB  conda-forge
#44 178.5     certifi-2019.11.28         |   py27h8c360ce_1         149 KB  conda-forge
#44 178.5     ld_impl_linux-64-2.40      |       h41732ed_0         688 KB  conda-forge
#44 178.5     libffi-3.2.1               |    he1b5a44_1007          47 KB  conda-forge
#44 178.5     libgcc-ng-13.1.0           |       he5830b7_0         758 KB  conda-forge
#44 178.5     libgomp-13.1.0             |       he5830b7_0         409 KB  conda-forge
#44 178.5     libsqlite-3.42.0           |       h2797004_0         809 KB  conda-forge
#44 178.5     libstdcxx-ng-13.1.0        |       hfd8a6a1_0         3.7 MB  conda-forge
#44 178.5     libzlib-1.2.13             |       hd590300_5          60 KB  conda-forge
#44 178.5     ncurses-6.4                |       hcb278e6_0         860 KB  conda-forge
#44 178.5     openssl-1.1.1v             |       hd590300_0         1.9 MB  conda-forge
#44 178.5     pip-20.1.1                 |     pyh9f0ad1d_0         1.1 MB  conda-forge
#44 178.5     python-2.7.15              |h5a48372_1011_cpython        12.2 MB  conda-forge
#44 178.5     python_abi-2.7             |         1_cp27mu           4 KB  conda-forge
#44 178.5     readline-8.2               |       h8228510_1         275 KB  conda-forge
#44 178.5     setuptools-44.0.0          |           py27_0         663 KB  conda-forge
#44 178.5     sqlite-3.42.0              |       h2c6b66d_0         799 KB  conda-forge
#44 178.5     tk-8.6.12                  |       h27826a3_0         3.3 MB  conda-forge
#44 178.5     wheel-0.37.1               |     pyhd8ed1ab_0          31 KB  conda-forge
#44 178.5     zlib-1.2.13                |       hd590300_5          91 KB  conda-forge
#44 178.5     ------------------------------------------------------------
#44 178.5                                            Total:        27.8 MB
#44 178.5 
#44 178.5 The following NEW packages will be INSTALLED:
#44 178.5 
#44 178.5   _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
#44 178.5   _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
#44 178.5   ca-certificates    conda-forge/linux-64::ca-certificates-2023.7.22-hbcca054_0 
#44 178.5   certifi            conda-forge/linux-64::certifi-2019.11.28-py27h8c360ce_1 
#44 178.5   ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0 
#44 178.5   libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007 
#44 178.5   libgcc-ng          conda-forge/linux-64::libgcc-ng-13.1.0-he5830b7_0 
#44 178.5   libgomp            conda-forge/linux-64::libgomp-13.1.0-he5830b7_0 
#44 178.5   libsqlite          conda-forge/linux-64::libsqlite-3.42.0-h2797004_0 
#44 178.5   libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.1.0-hfd8a6a1_0 
#44 178.5   libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5 
#44 178.5   ncurses            conda-forge/linux-64::ncurses-6.4-hcb278e6_0 
#44 178.5   openssl            conda-forge/linux-64::openssl-1.1.1v-hd590300_0 
#44 178.5   pip                conda-forge/noarch::pip-20.1.1-pyh9f0ad1d_0 
#44 178.5   python             conda-forge/linux-64::python-2.7.15-h5a48372_1011_cpython 
#44 178.5   python_abi         conda-forge/linux-64::python_abi-2.7-1_cp27mu 
#44 178.5   readline           conda-forge/linux-64::readline-8.2-h8228510_1 
#44 178.5   setuptools         conda-forge/linux-64::setuptools-44.0.0-py27_0 
#44 178.5   sqlite             conda-forge/linux-64::sqlite-3.42.0-h2c6b66d_0 
#44 178.5   tk                 conda-forge/linux-64::tk-8.6.12-h27826a3_0 
#44 178.5   wheel              conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0 
#44 178.5   zlib               conda-forge/linux-64::zlib-1.2.13-hd590300_5 
...
#44 787.8 Processing /home/rstudio/R/library/crisprScore/python/azimuth
#44 787.9 Requirement already satisfied: scipy<=1.2.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.2.1)
#44 787.9 Requirement already satisfied: numpy<=1.14 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.11.3)
#44 787.9 Requirement already satisfied: matplotlib in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (2.2.3)
#44 788.0 Requirement already satisfied: nose in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.3.7)
#44 788.0 Requirement already satisfied: scikit-learn<0.18,>=0.17.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (0.17.1)
#44 788.0 Requirement already satisfied: pandas in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (0.23.4)
#44 788.0 Requirement already satisfied: biopython<=1.76 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from Azimuth==2.0) (1.74)
#44 788.0 Requirement already satisfied: cycler>=0.10 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (0.10.0)
#44 788.0 Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2.4.7)
#44 788.0 Requirement already satisfied: python-dateutil>=2.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2.8.1)
#44 788.0 Requirement already satisfied: pytz in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (2020.1)
#44 788.0 Requirement already satisfied: six>=1.10 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.15.0)
#44 788.0 Requirement already satisfied: kiwisolver>=1.0.1 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.1.0)
#44 788.0 Requirement already satisfied: backports.functools_lru_cache in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (1.6.1)
#44 788.0 Requirement already satisfied: subprocess32 in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from matplotlib->Azimuth==2.0) (3.5.4)
#44 788.0 Requirement already satisfied: setuptools in /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/python2.7/site-packages (from kiwisolver>=1.0.1->matplotlib->Azimuth==2.0) (44.0.0.post20200102)
#44 788.0 Building wheels for collected packages: Azimuth
#44 788.0   Building wheel for Azimuth (setup.py): started
#44 788.2   Building wheel for Azimuth (setup.py): finished with status 'done'
#44 788.2   Created wheel for Azimuth: filename=Azimuth-2.0-py2-none-any.whl size=163884 sha256=cb56e2d7a8bad3018036c51eaa65e25109f5668f4c195acd3477d5618d491688
#44 788.2   Stored in directory: /tmp/pip-ephem-wheel-cache-KOxihB/wheels/56/4b/66/e075652909d01dd3d766e57ba0d9a334f4ddbcce926d80fe70
#44 788.2 Successfully built Azimuth
#44 788.2 Installing collected packages: Azimuth
#44 788.3 Successfully installed Azimuth-2.0
#44 789.3 Error: PyType_GetFlags - /home/rstudio/.cache/R/basilisk/1.13.3/crisprScore/1.3.4/azimuth_basilisk/lib/libpython2.7.so: undefined symbol: PyType_GetFlags
#44 789.3 In addition: Warning message:
#44 789.3 In py_initialize(config$python, config$libpython, config$pythonhome,  :
#44 789.3   Python 2 reached EOL on January 1, 2020. Python 2 compatability will be removed in an upcoming reticulate release.
#44 789.3 Execution halted
#44 ERROR: process "/bin/sh -c cd \"$SOFT/crisprScore\" && R -e \"library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)\"" did not complete successfully: exit code: 1

Maybe this is a reason: https://foss.heptapod.net/pypy/pypy/-/issues/3173#note_160826 ?

I suppose, that some versions should be updated maybe here: https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L1

LTLA commented 11 months ago

Is this the most recent crisprScore (1.4 release, 1.5 devel)?

I thought that the latest version was no longer using reticulate to load Python2 environments.

serge2016 commented 11 months ago

I have tried both the release and the latest master

LTLA commented 11 months ago

Oh wait, I see you're actually calling basiliskStart(env_azimuth). You shouldn't be doing that; from the output, you can see that reticulate's support for Python 2 is on its last legs. Instead, getAzimuthScores just uses the Conda environment's Python directly.

Incidentally, @Jfortin1 it may be safer to be more explicit about the Python used in

https://github.com/crisprVerse/crisprScore/blob/0bef9c4fce4e9d505bcbd847054dad1f02a0bc4f/R/getAzimuthScores.R#L86

e.g., with the full path via basilisk.utils:::getPythonBinary(env).

serge2016 commented 11 months ago

According to this repo, the crisprScore's release is 1.1.17. Master branch is newer, that's why I have tested it too.

With basiliskStart(env_azimuth) I simply try to install all needed environment. I am creating a docker image and want all installation to be done during build step for not to wait for the environment installation every run.

serge2016 commented 11 months ago

When I ran the same commands in the beginning of 2023 year, everything worked. I suppose the problem is in https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L184 or https://github.com/crisprVerse/crisprScore/blob/master/R/basilisk.R#L1

LTLA commented 11 months ago

To install up-front, you should set the BASILISK_USE_SYSTEM_DIR environment variable, as documented here.

Behavior at the start of the year doesn't really mean much, given reticulate has updated a few times since then (possibly dropping Python 2 support along the way) and conda itself seems to rebuild its binaries every now and then.

Jfortin1 commented 11 months ago

FIxed now in v1.5.1

serge2016 commented 11 months ago

If I use BASILISK_USE_SYSTEM_DIR=1, problem looks different:

 > library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)
#44 1.672 Error in obtainEnvironmentPath(env) : 
#44 1.672   environment 'azimuth_basilisk' should have been created during 'crisprScore' installation
#44 1.672 Calls: basiliskStart -> obtainEnvironmentPath
#44 1.672 Execution halted
serge2016 commented 11 months ago
> library(basilisk); source('R/basilisk.R'); obtainEnvironmentPath(env_azimuth)
Error in obtainEnvironmentPath(env_azimuth) :
  environment 'azimuth_basilisk' should have been created during 'crisprScore' installation
Execution halted
LTLA commented 11 months ago

I don't know how you're setting up your Dockerfile but you need to set this variable before installation of basilisk and crisprScore.

serge2016 commented 10 months ago

Thank you! Now I understood, that I should set ENV VAR BASILISK_USE_SYSTEM_DIR=1 before installing basilisk: R -e "remotes::install_github('LTLA/basilisk', ref='master'); remotes::install_github('LTLA/basilisk.utils', ref='master')"! After this I should install crisprVerse: R -e "remotes::install_github('crisprVerse/crisprBase', ref='master'); remotes::install_github('crisprVerse/crisprBowtie', ref='master'); remotes::install_github('crisprVerse/crisprScoreData', ref='master'); remotes::install_github('crisprVerse/crisprBwa', ref='master'); remotes::install_github('crisprVerse/crisprDesign', ref='master'); remotes::install_github('crisprVerse/crisprDesignData', ref='master'); remotes::install_github('crisprVerse/crisprViz', ref='master'); remotes::install_github('crisprVerse/crisprVerse', ref='main'); remotes::install_github('crisprVerse/crisprScore', ref='master')".

And finally I should NOT run cd "$SOFT/crisprScore" && R -e "library(basilisk); source('R/basilisk.R'); basiliskStart(env_azimuth)", it is not necessary (and fails with error).

Thank you! It works!