Closed khouakhi closed 4 years ago
@khouakhi Is it possible for you to use Miniconda 4.7.12?
wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh -O Miniconda3-4.7.12.1-Linux-x86_64.sh
bash ./Miniconda3-4.7.12.1-Linux-x86_64.sh -b -p miniconda3
source miniconda3/etc/profile.d/conda.sh
conda activate base
# Please pick up an unofficial release of cdat 8.2 on Linux
conda create -n cdat82 -c cdat/label/v82 -c conda-forge "cdat=8.2" "python=3.7" "numpy=1.17" mesalib="18.3.1" jupyter notebook ipywidgets
conda activate cdat82
# Note that I ran into problem when using latest miniconda3 on macos, I am not sure about # Linux, but we know for sure Miniconda 4.7.12 works with cdat 82 on both Linux and MacOS.
I managed to install cdms2 and cdutil but I could not import the modules Colab notebook here
!wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh !chmod +x Miniconda3-4.7.12.1-Linux-x86_64.sh !bash Miniconda3-4.7.12.1-Linux-x86_64.sh -b -f -p /usr/local import sys sys.path.append('/usr/local/lib/python3.6/site-packages/')
!conda install -q -y --prefix /usr/local cdms2 cdutil -c conda-forge
`The following packages will be downloaded:
package | build
---------------------------|-----------------
attrs-19.3.0 | py_0 35 KB conda-forge
bzip2-1.0.8 | h516909a_2 396 KB conda-forge
ca-certificates-2020.4.5.2 | hecda079_0 147 KB conda-forge
cdat_info-8.2 | py_7 16 KB conda-forge
cdms2-3.1.4 | py37ha6f5e91_3 677 KB conda-forge
cdtime-3.1.3 | py37h006a570_3 55 KB conda-forge
cdutil-8.2 | py_2 451 KB conda-forge
certifi-2020.4.5.2 | py37hc8dfbb8_0 152 KB conda-forge
conda-4.8.3 | py37hc8dfbb8_1 3.0 MB conda-forge
curl-7.69.1 | h33f0ec9_0 137 KB conda-forge
decorator-4.4.2 | py_0 11 KB conda-forge
distarray-2.12.2 | py_1 10 KB conda-forge
esmf-8.0.1 | nompi_hbeb3ca6_0 25.2 MB conda-forge
esmpy-8.0.1 |nompi_py37h777d1d2_0 169 KB conda-forge
expat-2.2.9 | he1b5a44_2 191 KB conda-forge
future-0.18.2 | py37hc8dfbb8_1 712 KB conda-forge
g2clib-1.6.0 | hf3f1b0b_9 56 KB conda-forge
genutil-8.2 | py37hc1659b7_3 132 KB conda-forge
hdf4-4.2.13 | hf30be14_1003 964 KB conda-forge
hdf5-1.10.6 |nompi_h3c11f04_100 3.0 MB conda-forge
importlib-metadata-1.6.1 | py37hc8dfbb8_0 44 KB conda-forge
importlib_metadata-1.6.1 | 0 3 KB conda-forge
ipython_genutils-0.2.0 | py_1 21 KB conda-forge
jasper-1.900.1 | h07fcdf6_1006 286 KB conda-forge
jpeg-9d | h516909a_0 266 KB conda-forge
jsonschema-3.2.0 | py37hc8dfbb8_1 89 KB conda-forge
jupyter_core-4.6.3 | py37hc8dfbb8_1 71 KB conda-forge
krb5-1.17.1 | h173b8e3_0 1.3 MB
lazy-object-proxy-1.5.0 | py37h8f50634_0 28 KB conda-forge
libblas-3.8.0 | 11_openblas 10 KB conda-forge
libcblas-3.8.0 | 11_openblas 10 KB conda-forge
libcdms-3.1.2 | h054cd8a_112 1.5 MB conda-forge
libcf-1.0.3 | py37hda0e254_108 254 KB conda-forge
libcurl-7.69.1 | hf7181ac_0 573 KB conda-forge
libdrs-3.1.2 | hc2e2db3_112 526 KB conda-forge
libdrs_f-3.1.2 | hae7e664_110 534 KB conda-forge
libgfortran-ng-7.5.0 | hdf63c60_6 1.7 MB conda-forge
liblapack-3.8.0 | 11_openblas 10 KB conda-forge
libnetcdf-4.7.4 |nompi_h84807e1_104 1.3 MB conda-forge
libopenblas-0.3.6 | h6e990d7_6 7.7 MB conda-forge
libpng-1.6.37 | hed695b0_1 308 KB conda-forge
libssh2-1.9.0 | hab1572f_2 298 KB conda-forge
libtiff-4.1.0 | hc7e4089_6 668 KB conda-forge
libuuid-2.32.1 | h14c3975_1000 26 KB conda-forge
libwebp-base-1.1.0 | h516909a_3 845 KB conda-forge
lz4-c-1.8.3 | he1b5a44_1001 187 KB conda-forge
nbformat-5.0.6 | py_0 99 KB conda-forge
netcdf-fortran-4.5.2 |nompi_hfef6a68_106 1.2 MB conda-forge
numpy-1.18.5 | py37h8960a57_0 5.1 MB conda-forge
openblas-0.3.6 | h6e990d7_6 8.2 MB conda-forge
openssl-1.1.1g | h516909a_0 2.1 MB conda-forge
pyrsistent-0.16.0 | py37h8f50634_0 89 KB conda-forge
python_abi-3.7 | 1_cp37m 4 KB conda-forge
traitlets-4.3.3 | py37hc8dfbb8_1 133 KB conda-forge
udunits2-2.2.27.6 | h4e0c4b3_1001 152 KB conda-forge
zipp-3.1.0 | py_0 10 KB conda-forge
zstd-1.4.4 | h3b9ef0a_2 982 KB conda-forge
------------------------------------------------------------
Total: 71.9 MB
The following NEW packages will be INSTALLED:
attrs conda-forge/noarch::attrs-19.3.0-py_0 bzip2 conda-forge/linux-64::bzip2-1.0.8-h516909a_2 cdat_info conda-forge/noarch::cdat_info-8.2-py_7 cdms2 conda-forge/linux-64::cdms2-3.1.4-py37ha6f5e91_3 cdtime conda-forge/linux-64::cdtime-3.1.3-py37h006a570_3 cdutil conda-forge/noarch::cdutil-8.2-py_2 curl conda-forge/linux-64::curl-7.69.1-h33f0ec9_0 decorator conda-forge/noarch::decorator-4.4.2-py_0 distarray conda-forge/noarch::distarray-2.12.2-py_1 esmf conda-forge/linux-64::esmf-8.0.1-nompi_hbeb3ca6_0 esmpy conda-forge/linux-64::esmpy-8.0.1-nompi_py37h777d1d2_0 expat conda-forge/linux-64::expat-2.2.9-he1b5a44_2 future conda-forge/linux-64::future-0.18.2-py37hc8dfbb8_1 g2clib conda-forge/linux-64::g2clib-1.6.0-hf3f1b0b_9 genutil conda-forge/linux-64::genutil-8.2-py37hc1659b7_3 hdf4 conda-forge/linux-64::hdf4-4.2.13-hf30be14_1003 hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_100 importlib-metadata conda-forge/linux-64::importlib-metadata-1.6.1-py37hc8dfbb8_0 importlib_metadata conda-forge/noarch::importlib_metadata-1.6.1-0 ipython_genutils conda-forge/noarch::ipython_genutils-0.2.0-py_1 jasper conda-forge/linux-64::jasper-1.900.1-h07fcdf6_1006 jpeg conda-forge/linux-64::jpeg-9d-h516909a_0 jsonschema conda-forge/linux-64::jsonschema-3.2.0-py37hc8dfbb8_1 jupyter_core conda-forge/linux-64::jupyter_core-4.6.3-py37hc8dfbb8_1 krb5 pkgs/main/linux-64::krb5-1.17.1-h173b8e3_0 lazy-object-proxy conda-forge/linux-64::lazy-object-proxy-1.5.0-py37h8f50634_0 libblas conda-forge/linux-64::libblas-3.8.0-11_openblas libcblas conda-forge/linux-64::libcblas-3.8.0-11_openblas libcdms conda-forge/linux-64::libcdms-3.1.2-h054cd8a_112 libcf conda-forge/linux-64::libcf-1.0.3-py37hda0e254_108 libcurl conda-forge/linux-64::libcurl-7.69.1-hf7181ac_0 libdrs conda-forge/linux-64::libdrs-3.1.2-hc2e2db3_112 libdrs_f conda-forge/linux-64::libdrs_f-3.1.2-hae7e664_110 libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-hdf63c60_6 liblapack conda-forge/linux-64::liblapack-3.8.0-11_openblas libnetcdf conda-forge/linux-64::libnetcdf-4.7.4-nompi_h84807e1_104 libopenblas conda-forge/linux-64::libopenblas-0.3.6-h6e990d7_6 libpng conda-forge/linux-64::libpng-1.6.37-hed695b0_1 libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_2 libtiff conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6 libuuid conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000 libwebp-base conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3 lz4-c conda-forge/linux-64::lz4-c-1.8.3-he1b5a44_1001 nbformat conda-forge/noarch::nbformat-5.0.6-py_0 netcdf-fortran conda-forge/linux-64::netcdf-fortran-4.5.2-nompi_hfef6a68_106 numpy conda-forge/linux-64::numpy-1.18.5-py37h8960a57_0 openblas conda-forge/linux-64::openblas-0.3.6-h6e990d7_6 pyrsistent conda-forge/linux-64::pyrsistent-0.16.0-py37h8f50634_0 python_abi conda-forge/linux-64::python_abi-3.7-1_cp37m traitlets conda-forge/linux-64::traitlets-4.3.3-py37hc8dfbb8_1 udunits2 conda-forge/linux-64::udunits2-2.2.27.6-h4e0c4b3_1001 zipp conda-forge/noarch::zipp-3.1.0-py_0 zstd conda-forge/linux-64::zstd-1.4.4-h3b9ef0a_2
The following packages will be UPDATED:
ca-certificates pkgs/main::ca-certificates-2019.10.16~ --> conda-forge::ca-certificates-2020.4.5.2-hecda079_0 certifi pkgs/main::certifi-2019.9.11-py37_0 --> conda-forge::certifi-2020.4.5.2-py37hc8dfbb8_0 conda pkgs/main::conda-4.7.12-py37_0 --> conda-forge::conda-4.8.3-py37hc8dfbb8_1 openssl pkgs/main::openssl-1.1.1d-h7b6447c_3 --> conda-forge::openssl-1.1.1g-h516909a_0
Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done
import cdms2
I cannot tell why you cannot import cdms2. Would you be willing to try creating a separate environment just like I showed above?
Installed CDAT and 'import cv2' works but
import cdms2
ImportError Traceback (most recent call last)
@khouakhi The mixing of these two environments, colab and conda, is not something we support. It can create undesirable behavior. In your colab environment, cdms2 works well when running using the python it was installed with, using it any other way than this is not supported. The following demonstrates cdms2
working correctly using the python it was installed with.
%%bash
# Numpy was missing
conda install -c conda-forge "numpy=1.17"
/usr/local/bin/python -c "import cdms2; print(cdms2);
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc');
print(f.variables)"
At this time the only recommendation we can make to continue using colab is the following example.
%%bash
cat << EOF > test.py
import cdms2
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc')
print(f.variables)
EOF
/usr/local/bin/python test.py
@jasonb5 I added numpy as suggested and used cat << EOF > test.py not I understand your demo
!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh !chmod +x Miniconda3-latest-Linux-x86_64.sh !bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
!conda install -c conda-forge "numpy=1.17" cdms2
cat << EOF > test.py import cdms2 /bin/bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOF')
@khouakhi Looks like you might be missing the ending line with EOF
cat << EOF > test.py
import cdms2
EOF
@khouakhi Please reopen if you still need assistance.
I tried to install cdms2 in conda from Colab and I get
Executing transaction: ...working... failed ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'. NotADirectoryError(20, 'Not a directory') Attempting to roll back. Rolling back transaction: ...working... done
details below:
`!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh !chmod +x Miniconda3-latest-Linux-x86_64.sh !bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
!conda install -c conda-forge cdms2`
Solving environment: ...working... done
Package Plan
environment location: /usr/bin
added / updated specs:
The following NEW packages will be INSTALLED:
Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... failed ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'. NotADirectoryError(20, 'Not a directory') Attempting to roll back.
Rolling back transaction: ...working... done
NotADirectoryError(20, 'Not a directory')