search

CDAT / cdat

Community Data Analysis Tools
Other
174 stars 68 forks source link

install from google colab #2253

Closed khouakhi closed 4 years ago

khouakhi commented 4 years ago

I tried to install cdms2 in conda from Colab and I get

Executing transaction: ...working... failed ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'. NotADirectoryError(20, 'Not a directory') Attempting to roll back. Rolling back transaction: ...working... done

details below:

`!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh !chmod +x Miniconda3-latest-Linux-x86_64.sh !bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

!conda install -c conda-forge cdms2`

Solving environment: ...working... done

Package Plan

environment location: /usr/bin

added / updated specs:

The following NEW packages will be INSTALLED:

_libgcc_mutex:      0.1-conda_forge              conda-forge
_openmp_mutex:      4.5-0_gnu                    conda-forge
attrs:              19.3.0-py_0                  conda-forge
brotlipy:           0.7.0-py36h8c4c3a4_1000      conda-forge
bzip2:              1.0.8-h516909a_2             conda-forge
ca-certificates:    2020.4.5.2-hecda079_0        conda-forge
cdat_info:          8.2-py_7                     conda-forge
cdms2:              3.1.4-py36h25070a4_10        conda-forge
cdtime:             3.1.3-py36h76a73ab_3         conda-forge
certifi:            2020.4.5.2-py36h9f0ad1d_0    conda-forge
cffi:               1.14.0-py36hd463f26_0        conda-forge
chardet:            3.0.4-py36h9f0ad1d_1006      conda-forge
cryptography:       2.9.2-py36h45558ae_0         conda-forge
curl:               7.69.1-h33f0ec9_0            conda-forge
decorator:          4.4.2-py_0                   conda-forge
distarray:          2.12.2-py_1                  conda-forge
esmf:               8.0.1-nompi_hbeb3ca6_0       conda-forge
esmpy:              8.0.1-nompi_py36hbd5f7c1_0   conda-forge
future:             0.18.2-py36h9f0ad1d_1        conda-forge
g2clib:             1.6.0-hf3f1b0b_9             conda-forge
hdf4:               4.2.13-hf30be14_1003         conda-forge
hdf5:               1.10.6-nompi_h3c11f04_100    conda-forge
idna:               2.9-py_1                     conda-forge
importlib-metadata: 1.6.1-py36h9f0ad1d_0         conda-forge
importlib_metadata: 1.6.1-0                      conda-forge
ipython_genutils:   0.2.0-py_1                   conda-forge
jasper:             1.900.1-h07fcdf6_1006        conda-forge
jpeg:               9d-h516909a_0                conda-forge
jsonschema:         3.2.0-py36h9f0ad1d_1         conda-forge
jupyter_core:       4.6.3-py36h9f0ad1d_1         conda-forge
krb5:               1.17.1-h2fd8d38_0            conda-forge
lazy-object-proxy:  1.5.0-py36h8c4c3a4_0         conda-forge
ld_impl_linux-64:   2.34-h53a641e_5              conda-forge
libblas:            3.8.0-11_openblas            conda-forge
libcblas:           3.8.0-11_openblas            conda-forge
libcdms:            3.1.2-h054cd8a_112           conda-forge
libcf:              1.0.3-py36ha507d3b_108       conda-forge
libcurl:            7.69.1-hf7181ac_0            conda-forge
libdrs:             3.1.2-hc2e2db3_112           conda-forge
libdrs_f:           3.1.2-hae7e664_110           conda-forge
libedit:            3.1.20191231-h46ee950_0      conda-forge
libffi:             3.2.1-he1b5a44_1007          conda-forge
libgcc-ng:          9.2.0-h24d8f2e_2             conda-forge
libgfortran-ng:     7.5.0-hdf63c60_6             conda-forge
libgomp:            9.2.0-h24d8f2e_2             conda-forge
liblapack:          3.8.0-11_openblas            conda-forge
libnetcdf:          4.7.4-nompi_h84807e1_104     conda-forge
libopenblas:        0.3.6-h6e990d7_6             conda-forge
libpng:             1.6.37-hed695b0_1            conda-forge
libssh2:            1.9.0-hab1572f_2             conda-forge
libstdcxx-ng:       9.2.0-hdf63c60_2             conda-forge
libtiff:            4.1.0-hc7e4089_6             conda-forge
libuuid:            2.32.1-h14c3975_1000         conda-forge
libwebp-base:       1.1.0-h516909a_3             conda-forge
lz4-c:              1.9.2-he1b5a44_1             conda-forge
nbformat:           5.0.6-py_0                   conda-forge
ncurses:            6.1-hf484d3e_1002            conda-forge
netcdf-fortran:     4.5.2-nompi_hfef6a68_106     conda-forge
numpy:              1.18.5-py36h7314795_0        conda-forge
openblas:           0.3.6-h6e990d7_6             conda-forge
openssl:            1.1.1g-h516909a_0            conda-forge
pip:                20.1.1-py_1                  conda-forge
pycparser:          2.20-py_0                    conda-forge
pyopenssl:          19.1.0-py_1                  conda-forge
pyrsistent:         0.16.0-py36h8c4c3a4_0        conda-forge
pysocks:            1.7.1-py36h9f0ad1d_1         conda-forge
python:             3.6.10-h8356626_1011_cpython conda-forge
python_abi:         3.6-1_cp36m                  conda-forge
readline:           8.0-hf8c457e_0               conda-forge
requests:           2.23.0-pyh8c360ce_2          conda-forge
setuptools:         47.1.1-py36h9f0ad1d_0        conda-forge
six:                1.15.0-pyh9f0ad1d_0          conda-forge
sqlite:             3.30.1-hcee41ef_0            conda-forge
tk:                 8.6.10-hed695b0_0            conda-forge
traitlets:          4.3.3-py36h9f0ad1d_1         conda-forge
urllib3:            1.25.9-py_0                  conda-forge
wheel:              0.34.2-py_1                  conda-forge
xz:                 5.2.5-h516909a_0             conda-forge
zipp:               3.1.0-py_0                   conda-forge
zlib:               1.2.11-h516909a_1006         conda-forge
zstd:               1.4.4-h6597ccf_3             conda-forge

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... failed ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'. NotADirectoryError(20, 'Not a directory') Attempting to roll back.

Rolling back transaction: ...working... done

NotADirectoryError(20, 'Not a directory')

muryanto1 commented 4 years ago

@khouakhi Is it possible for you to use Miniconda 4.7.12?

wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh -O Miniconda3-4.7.12.1-Linux-x86_64.sh
bash ./Miniconda3-4.7.12.1-Linux-x86_64.sh -b -p miniconda3
source miniconda3/etc/profile.d/conda.sh
conda activate base

# Please pick up an unofficial release of cdat 8.2 on Linux
conda create -n cdat82 -c cdat/label/v82 -c conda-forge "cdat=8.2" "python=3.7" "numpy=1.17" mesalib="18.3.1" jupyter notebook ipywidgets
conda activate cdat82

# Note that I ran into problem when using latest miniconda3 on macos, I am not sure about # Linux, but we know for sure Miniconda 4.7.12 works with cdat 82 on both Linux and MacOS.
khouakhi commented 4 years ago

I managed to install cdms2 and cdutil but I could not import the modules Colab notebook here

!wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh !chmod +x Miniconda3-4.7.12.1-Linux-x86_64.sh !bash Miniconda3-4.7.12.1-Linux-x86_64.sh -b -f -p /usr/local import sys sys.path.append('/usr/local/lib/python3.6/site-packages/')

!conda install -q -y --prefix /usr/local cdms2 cdutil -c conda-forge

`The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
attrs-19.3.0               |             py_0          35 KB  conda-forge
bzip2-1.0.8                |       h516909a_2         396 KB  conda-forge
ca-certificates-2020.4.5.2 |       hecda079_0         147 KB  conda-forge
cdat_info-8.2              |             py_7          16 KB  conda-forge
cdms2-3.1.4                |   py37ha6f5e91_3         677 KB  conda-forge
cdtime-3.1.3               |   py37h006a570_3          55 KB  conda-forge
cdutil-8.2                 |             py_2         451 KB  conda-forge
certifi-2020.4.5.2         |   py37hc8dfbb8_0         152 KB  conda-forge
conda-4.8.3                |   py37hc8dfbb8_1         3.0 MB  conda-forge
curl-7.69.1                |       h33f0ec9_0         137 KB  conda-forge
decorator-4.4.2            |             py_0          11 KB  conda-forge
distarray-2.12.2           |             py_1          10 KB  conda-forge
esmf-8.0.1                 | nompi_hbeb3ca6_0        25.2 MB  conda-forge
esmpy-8.0.1                |nompi_py37h777d1d2_0         169 KB  conda-forge
expat-2.2.9                |       he1b5a44_2         191 KB  conda-forge
future-0.18.2              |   py37hc8dfbb8_1         712 KB  conda-forge
g2clib-1.6.0               |       hf3f1b0b_9          56 KB  conda-forge
genutil-8.2                |   py37hc1659b7_3         132 KB  conda-forge
hdf4-4.2.13                |    hf30be14_1003         964 KB  conda-forge
hdf5-1.10.6                |nompi_h3c11f04_100         3.0 MB  conda-forge
importlib-metadata-1.6.1   |   py37hc8dfbb8_0          44 KB  conda-forge
importlib_metadata-1.6.1   |                0           3 KB  conda-forge
ipython_genutils-0.2.0     |             py_1          21 KB  conda-forge
jasper-1.900.1             |    h07fcdf6_1006         286 KB  conda-forge
jpeg-9d                    |       h516909a_0         266 KB  conda-forge
jsonschema-3.2.0           |   py37hc8dfbb8_1          89 KB  conda-forge
jupyter_core-4.6.3         |   py37hc8dfbb8_1          71 KB  conda-forge
krb5-1.17.1                |       h173b8e3_0         1.3 MB
lazy-object-proxy-1.5.0    |   py37h8f50634_0          28 KB  conda-forge
libblas-3.8.0              |      11_openblas          10 KB  conda-forge
libcblas-3.8.0             |      11_openblas          10 KB  conda-forge
libcdms-3.1.2              |     h054cd8a_112         1.5 MB  conda-forge
libcf-1.0.3                | py37hda0e254_108         254 KB  conda-forge
libcurl-7.69.1             |       hf7181ac_0         573 KB  conda-forge
libdrs-3.1.2               |     hc2e2db3_112         526 KB  conda-forge
libdrs_f-3.1.2             |     hae7e664_110         534 KB  conda-forge
libgfortran-ng-7.5.0       |       hdf63c60_6         1.7 MB  conda-forge
liblapack-3.8.0            |      11_openblas          10 KB  conda-forge
libnetcdf-4.7.4            |nompi_h84807e1_104         1.3 MB  conda-forge
libopenblas-0.3.6          |       h6e990d7_6         7.7 MB  conda-forge
libpng-1.6.37              |       hed695b0_1         308 KB  conda-forge
libssh2-1.9.0              |       hab1572f_2         298 KB  conda-forge
libtiff-4.1.0              |       hc7e4089_6         668 KB  conda-forge
libuuid-2.32.1             |    h14c3975_1000          26 KB  conda-forge
libwebp-base-1.1.0         |       h516909a_3         845 KB  conda-forge
lz4-c-1.8.3                |    he1b5a44_1001         187 KB  conda-forge
nbformat-5.0.6             |             py_0          99 KB  conda-forge
netcdf-fortran-4.5.2       |nompi_hfef6a68_106         1.2 MB  conda-forge
numpy-1.18.5               |   py37h8960a57_0         5.1 MB  conda-forge
openblas-0.3.6             |       h6e990d7_6         8.2 MB  conda-forge
openssl-1.1.1g             |       h516909a_0         2.1 MB  conda-forge
pyrsistent-0.16.0          |   py37h8f50634_0          89 KB  conda-forge
python_abi-3.7             |          1_cp37m           4 KB  conda-forge
traitlets-4.3.3            |   py37hc8dfbb8_1         133 KB  conda-forge
udunits2-2.2.27.6          |    h4e0c4b3_1001         152 KB  conda-forge
zipp-3.1.0                 |             py_0          10 KB  conda-forge
zstd-1.4.4                 |       h3b9ef0a_2         982 KB  conda-forge
------------------------------------------------------------
                                       Total:        71.9 MB

The following NEW packages will be INSTALLED:

attrs conda-forge/noarch::attrs-19.3.0-py_0 bzip2 conda-forge/linux-64::bzip2-1.0.8-h516909a_2 cdat_info conda-forge/noarch::cdat_info-8.2-py_7 cdms2 conda-forge/linux-64::cdms2-3.1.4-py37ha6f5e91_3 cdtime conda-forge/linux-64::cdtime-3.1.3-py37h006a570_3 cdutil conda-forge/noarch::cdutil-8.2-py_2 curl conda-forge/linux-64::curl-7.69.1-h33f0ec9_0 decorator conda-forge/noarch::decorator-4.4.2-py_0 distarray conda-forge/noarch::distarray-2.12.2-py_1 esmf conda-forge/linux-64::esmf-8.0.1-nompi_hbeb3ca6_0 esmpy conda-forge/linux-64::esmpy-8.0.1-nompi_py37h777d1d2_0 expat conda-forge/linux-64::expat-2.2.9-he1b5a44_2 future conda-forge/linux-64::future-0.18.2-py37hc8dfbb8_1 g2clib conda-forge/linux-64::g2clib-1.6.0-hf3f1b0b_9 genutil conda-forge/linux-64::genutil-8.2-py37hc1659b7_3 hdf4 conda-forge/linux-64::hdf4-4.2.13-hf30be14_1003 hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_100 importlib-metadata conda-forge/linux-64::importlib-metadata-1.6.1-py37hc8dfbb8_0 importlib_metadata conda-forge/noarch::importlib_metadata-1.6.1-0 ipython_genutils conda-forge/noarch::ipython_genutils-0.2.0-py_1 jasper conda-forge/linux-64::jasper-1.900.1-h07fcdf6_1006 jpeg conda-forge/linux-64::jpeg-9d-h516909a_0 jsonschema conda-forge/linux-64::jsonschema-3.2.0-py37hc8dfbb8_1 jupyter_core conda-forge/linux-64::jupyter_core-4.6.3-py37hc8dfbb8_1 krb5 pkgs/main/linux-64::krb5-1.17.1-h173b8e3_0 lazy-object-proxy conda-forge/linux-64::lazy-object-proxy-1.5.0-py37h8f50634_0 libblas conda-forge/linux-64::libblas-3.8.0-11_openblas libcblas conda-forge/linux-64::libcblas-3.8.0-11_openblas libcdms conda-forge/linux-64::libcdms-3.1.2-h054cd8a_112 libcf conda-forge/linux-64::libcf-1.0.3-py37hda0e254_108 libcurl conda-forge/linux-64::libcurl-7.69.1-hf7181ac_0 libdrs conda-forge/linux-64::libdrs-3.1.2-hc2e2db3_112 libdrs_f conda-forge/linux-64::libdrs_f-3.1.2-hae7e664_110 libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-hdf63c60_6 liblapack conda-forge/linux-64::liblapack-3.8.0-11_openblas libnetcdf conda-forge/linux-64::libnetcdf-4.7.4-nompi_h84807e1_104 libopenblas conda-forge/linux-64::libopenblas-0.3.6-h6e990d7_6 libpng conda-forge/linux-64::libpng-1.6.37-hed695b0_1 libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_2 libtiff conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6 libuuid conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000 libwebp-base conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3 lz4-c conda-forge/linux-64::lz4-c-1.8.3-he1b5a44_1001 nbformat conda-forge/noarch::nbformat-5.0.6-py_0 netcdf-fortran conda-forge/linux-64::netcdf-fortran-4.5.2-nompi_hfef6a68_106 numpy conda-forge/linux-64::numpy-1.18.5-py37h8960a57_0 openblas conda-forge/linux-64::openblas-0.3.6-h6e990d7_6 pyrsistent conda-forge/linux-64::pyrsistent-0.16.0-py37h8f50634_0 python_abi conda-forge/linux-64::python_abi-3.7-1_cp37m traitlets conda-forge/linux-64::traitlets-4.3.3-py37hc8dfbb8_1 udunits2 conda-forge/linux-64::udunits2-2.2.27.6-h4e0c4b3_1001 zipp conda-forge/noarch::zipp-3.1.0-py_0 zstd conda-forge/linux-64::zstd-1.4.4-h3b9ef0a_2

The following packages will be UPDATED:

ca-certificates pkgs/main::ca-certificates-2019.10.16~ --> conda-forge::ca-certificates-2020.4.5.2-hecda079_0 certifi pkgs/main::certifi-2019.9.11-py37_0 --> conda-forge::certifi-2020.4.5.2-py37hc8dfbb8_0 conda pkgs/main::conda-4.7.12-py37_0 --> conda-forge::conda-4.8.3-py37hc8dfbb8_1 openssl pkgs/main::openssl-1.1.1d-h7b6447c_3 --> conda-forge::openssl-1.1.1g-h516909a_0

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done

import cdms2

muryanto1 commented 4 years ago

I cannot tell why you cannot import cdms2. Would you be willing to try creating a separate environment just like I showed above?

khouakhi commented 4 years ago

Installed CDAT and 'import cv2' works but

import cdms2

ImportError Traceback (most recent call last)

in () ----> 1 import cdms2 1 frames /usr/local/lib/python3.7/site-packages/cdms2/dataset.py in () 6 from .error import CDMSError 7 import sys ----> 8 from . import Cdunif 9 import numpy 10 from . import cdmsNode ImportError: cannot import name 'Cdunif'
jasonb5 commented 4 years ago

@khouakhi The mixing of these two environments, colab and conda, is not something we support. It can create undesirable behavior. In your colab environment, cdms2 works well when running using the python it was installed with, using it any other way than this is not supported. The following demonstrates cdms2 working correctly using the python it was installed with.

%%bash
# Numpy was missing
conda install -c conda-forge "numpy=1.17"
/usr/local/bin/python -c "import cdms2; print(cdms2);
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc');
print(f.variables)"

At this time the only recommendation we can make to continue using colab is the following example.

%%bash
cat << EOF > test.py
import cdms2
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc')
print(f.variables)
EOF

/usr/local/bin/python test.py
khouakhi commented 4 years ago

@jasonb5 I added numpy as suggested and used cat << EOF > test.py not I understand your demo

get miniconda

!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh !chmod +x Miniconda3-latest-Linux-x86_64.sh !bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

conda install cdms2 with numpy 1.17

!conda install -c conda-forge "numpy=1.17" cdms2

import cdms2

cat << EOF > test.py import cdms2 /bin/bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOF')

jasonb5 commented 4 years ago

@khouakhi Looks like you might be missing the ending line with EOF

cat << EOF > test.py
import cdms2
EOF
jasonb5 commented 4 years ago

@khouakhi Please reopen if you still need assistance.