search

NCAR / pynio

PyNIO is a multi-format data I/O package with a NetCDF-style interface
http://www.pyngl.ucar.edu/Nio.shtml
Apache License 2.0
112 stars 37 forks source link

Python 3 support #10

Closed darothen closed 6 years ago

darothen commented 7 years ago

In support of upstream Python 3 compatibility (NCAR/PyReshaper#26).

scollis commented 7 years ago

Any progress on Py3.x support. Xarray is using Pynio as a ingest for grib... I am nearly 100% 3.x now and don't want to go back.

david-ian-brown commented 7 years ago

Hi Scott and Daniel, We are currently working on Python 3 support for PyNIO as a high priority task. The developer is currently optimistic that it won't take too long so hopefully it will be available fairly soon. -dave

On Mon, Jun 26, 2017 at 9:23 AM, Scott notifications@github.com wrote:

Any progress on Py3.x support. Xarray is using Pynio as a ingest for grib... I am nearly 100% 3.x now and don't want to go back.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NCAR/pynio/issues/10#issuecomment-311092301, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbWwlwJPMJwnQhDZtEgpuUNpkV797ks5sH8zlgaJpZM4LXoMx .

PeterDSteinberg commented 7 years ago

I'm working on an ML project for NASA (related to @darothen 's mention of earthio above). We're glad to hear Python 3 support is coming for pynio. Thanks!

darothen commented 7 years ago

@david-ian-brown any late summer updates on this? I have an application using HRRR GRIB2 output which leans heavily on xarray; if I could skip the step of converting GRIB2 to NetCDF (which would be possible with PyNIO support for Python 3), it would greatly help.

lukelbd commented 7 years ago

I'm also super excited to see pynio with python 3 support.

But for anyone reading this thread: in the meantime, if you use anaconda, you can easily create a python 2.7 virtual environment. This link has some great info. Create the environment with conda create -n python2 python=2.7 anaconda (where python2 is the name I chose for the virtualenv), install the relevant packages with conda install -n python2 xarray and conda install -n python2 -c conda-forge pynio, activate the environment from the command-line with source activate python2 or . activate python2, and launch an ipython terminal, jupyter notebook, etc while the environment is "activated."

This is useful for any other packages that haven't yet been ported to python 3.

darothen commented 7 years ago

@lukelbd that's true - but the scientific Python community is quickly moving away from Python 2, and there is little reason to maintain two fully functioning development environments for daily work. Virtual environments aren't a solution here in many cases. For instance, I'm building a workflow to ingest/parse GRIB2 output via xarray for work, and our entire stack is Python 3; it's hugely disruptive to maintain a separate Python stack on all our cloud services just to read those files into our system.

lukelbd commented 7 years ago

@darothen you're absolutely right; just wanted to share a band-aid solution (viable for not-too-intensive workflows) until the python 3 update arrives. Though it looks like that will be imminent, so maybe it was not necessary.

bladwig1 commented 7 years ago

Unfortunately, the PyNIO port is taking a little longer than I expected, so I wouldn't say that release is imminent. It's currently in the debug stage, soon approaching testing.

darothen commented 7 years ago

Thanks for hte update @bladwig1 . Is there anything the community to do at this stage to help out?

bladwig1 commented 6 years ago

PyNIO for Python 3.x is now merged in to the 'develop' branch on GitHub. However, an official release on conda-forge won't take place until a large amount of bug fixes related to compound data types, HDF5, etc are included. That being said, you should be able to pull down PyNIO from GitHub and start trying it out with Python 3.x.

bladwig1 commented 6 years ago

There is now a "dev" snapshot build on conda (ncar channel), which you can get by doing:

conda create -n (your_env_name) -c ncar -c conda-forge pynio=dev python=3

If you notice any issues, please create a new GitHub issue.

Thanks!

darothen commented 6 years ago

Hey all - I know this thread is technically closed, but there's an issue with this old protocol.

Somewhere along the line, a dependency issue was introduced. If you install a clean Python 3 environment with just xarray, a la:

$ conda create -n test_nio -c conda-forge python=3.6 xarray

then activate it and attempt to install pynio as referenced above:

$ conda install -c ncar pynio=dev

you'll run into an issue with libnetcdf versions, where it requests a downgrade:

The following packages will be DOWNGRADED:

    libnetcdf:     4.5.0-3                   conda-forge --> 4.4.1.1-10   conda-forge
    netcdf4:       1.3.1-py36_2              conda-forge --> 1.3.1-py36_1 conda-forge

Somewhere along the line, hdf5 gets messed up and it renders pynio un-importable. On my MacBook I'll get import errors such as:

ImportError: dlopen(/Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libhdf5_hl.10.dylib
  Referenced from: /Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so
  Reason: image not found

Any thoughts and/or solutions? This is new behavior; last time I helped someone get set-up with pynio/Python 3 back in January we did not run into this problem.

scollis commented 6 years ago

Yeah add me to the list of people this no longer works for..

ocefpaf commented 6 years ago

The issue here is a mismatch between the ncar channel and the conda-forge channel for the dependencies used ni both pynio and xarray. I recommend to release the dev version of pynio in a sub-label in conda-forge, that way people could do:

$ conda install --channel conda-forge/label/dev pynio

this would be in sync with the other dependencies we have in conda-forge and would avoid this issue.

khallock commented 6 years ago

Thanks for reporting this @darothen and @scollis, I'm looking into this now.

@ocefpaf, I will create an issue to further discuss your idea on the pynio-feedstock repo.

darothen commented 6 years ago

I can confirm that @ocefpaf's solution has worked just fine, and fixes the dependency issues.

ocefpaf commented 6 years ago

Thanks to @khallock who promptly created the dev version on conda-forge.

PS: he also created a ncl dev package too, install it with

conda install --channel conda-forge/label/dev ncl

:tada:

scollis commented 6 years ago

Thanks all!

-sent from a mobile device-

On Mar 18, 2018, at 1:57 PM, Filipe notifications@github.com wrote:

Thanks to @khallock who promptly created the dev version on conda-forge.

PS: he also created a ncl dev package too, install it with

conda install --channel conda-forge/label/dev ncl 🎉

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

darothen-cc commented 6 years ago

FWIW, there may be a new issue - the resolved method isn't working, and is forcing a downgrade to Python 2.7. I think I've traced this down to an older version of cyordereddict which is pinned to Python 2.7, and is a dependency of xarray.

ocefpaf commented 6 years ago

@darothen what happens if you specify the python version too? Like

conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio

The conda solver tries to wiggle its way out of complex dependencies by downgrading package version and build number, sometimes it even breaks channel preference! So the more you specify in the install command the better.

darothen-cc commented 6 years ago

@ocefpaf No dice, unfortunately:

image

ocefpaf commented 6 years ago

Works for me, do you have other channel in your .condarc?

> conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda

## Package Plan ##

  environment location: /home/filipe/miniconda3/envs/TEST

  added / updated specs: 
    - ipykernel
    - pip
    - pynio
    - python=3.6
    - xarray

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    krb5-1.14.6                |                0         4.0 MB  conda-forge
    backcall-0.1.0             |             py_0          13 KB  conda-forge
    distributed-1.21.5         |           py36_0         772 KB  conda-forge
    bokeh-0.12.15              |           py36_0         4.2 MB  conda-forge
    pynio-1.5.1.dev0           |           py36_1         4.3 MB  conda-forge/label/dev
    wheel-0.31.0               |           py36_0          62 KB  conda-forge
    msgpack-python-0.5.6       |           py36_0         300 KB  conda-forge
    ipython-6.3.0              |           py36_0         1.0 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        14.7 MB

The following NEW packages will be INSTALLED:

    backcall:         0.1.0-py_0            conda-forge          
    bokeh:            0.12.15-py36_0        conda-forge          
    boost:            1.66.0-py36_1         conda-forge          
    boost-cpp:        1.66.0-1              conda-forge          
    bottleneck:       1.2.1-py36_1          conda-forge          
    bzip2:            1.0.6-1               conda-forge          
    ca-certificates:  2018.1.18-0           conda-forge          
    cairo:            1.14.10-0             conda-forge          
    certifi:          2018.1.18-py36_0      conda-forge          
    click:            6.7-py_1              conda-forge          
    cloudpickle:      0.5.2-py_0            conda-forge          
    curl:             7.59.0-0              conda-forge          
    cytoolz:          0.9.0.1-py36_0        conda-forge          
    dask:             0.17.2-py_0           conda-forge          
    dask-core:        0.17.2-py_0           conda-forge          
    decorator:        4.2.1-py36_0          conda-forge          
    distributed:      1.21.5-py36_0         conda-forge          
    expat:            2.2.5-0               conda-forge          
    fontconfig:       2.12.6-0              conda-forge          
    freetype:         2.8.1-0               conda-forge          
    freexl:           1.0.5-0               conda-forge          
    g2clib:           1.6.0-5               conda-forge          
    geos:             3.6.2-1               conda-forge          
    geotiff:          1.4.2-1               conda-forge          
    gettext:          0.19.8.1-0            conda-forge          
    giflib:           5.1.4-0               conda-forge          
    glib:             2.55.0-0              conda-forge          
    h5netcdf:         0.5.0-py_0            conda-forge          
    h5py:             2.7.1-py36_2          conda-forge          
    hdf4:             4.2.13-0              conda-forge          
    hdf5:             1.10.1-2              conda-forge          
    hdfeos2:          2.19.1-8              conda-forge          
    hdfeos5:          5.1.16-0              conda-forge          
    heapdict:         1.0.0-py36_0          conda-forge          
    icu:              58.2-0                conda-forge          
    intel-openmp:     2018.0.0-8            defaults             
    ipykernel:        4.8.2-py36_0          conda-forge          
    ipython:          6.3.0-py36_0          conda-forge          
    ipython_genutils: 0.2.0-py36_0          conda-forge          
    jasper:           1.900.1-4             conda-forge          
    jedi:             0.11.1-py36_0         conda-forge          
    jinja2:           2.10-py36_0           conda-forge          
    jpeg:             9b-2                  conda-forge          
    json-c:           0.12.1-0              conda-forge          
    jupyter_client:   5.2.3-py36_0          conda-forge          
    jupyter_core:     4.4.0-py_0            conda-forge          
    kealib:           1.4.7-4               conda-forge          
    krb5:             1.14.6-0              conda-forge          
    libdap4:          3.18.3-2              conda-forge          
    libffi:           3.2.1-3               conda-forge          
    libgcc-ng:        7.2.0-hdf63c60_3      defaults             
    libgdal:          2.2.4-0               conda-forge          
    libgfortran:      3.0.0-1               defaults             
    libgfortran-ng:   7.2.0-hdf63c60_3      defaults             
    libiconv:         1.15-0                conda-forge          
    libkml:           1.3.0-6               conda-forge          
    libnetcdf:        4.4.1.1-10            conda-forge          
    libpng:           1.6.34-0              conda-forge          
    libpq:            9.6.3-0               conda-forge          
    libsodium:        1.0.16-0              conda-forge          
    libspatialite:    4.3.0a-19             conda-forge          
    libssh2:          1.8.0-2               conda-forge          
    libstdcxx-ng:     7.2.0-hdf63c60_3      defaults             
    libtiff:          4.0.9-0               conda-forge          
    libxml2:          2.9.8-0               conda-forge          
    locket:           0.2.0-py36_1          conda-forge          
    markupsafe:       1.0-py36_0            conda-forge          
    mkl:              2018.0.2-1            defaults             
    mkl_fft:          1.0.1-py36_1          conda-forge          
    mkl_random:       1.0.1-py36_0          conda-forge          
    msgpack-python:   0.5.6-py36_0          conda-forge          
    ncurses:          5.9-10                conda-forge          
    netcdf4:          1.3.1-py36_1          conda-forge          
    numpy:            1.14.2-py36hdbf6ddf_1 defaults             
    openjpeg:         2.3.0-2               conda-forge          
    openssl:          1.0.2n-0              conda-forge          
    packaging:        17.1-py_0             conda-forge          
    pandas:           0.22.0-py36_0         conda-forge          
    parso:            0.1.1-py_0            conda-forge          
    partd:            0.3.8-py36_0          conda-forge          
    pcre:             8.41-1                conda-forge          
    pexpect:          4.4.0-py36_0          conda-forge          
    pickleshare:      0.7.4-py36_0          conda-forge          
    pip:              9.0.3-py36_0          conda-forge          
    pixman:           0.34.0-1              conda-forge          
    poppler:          0.61.1-3              conda-forge          
    poppler-data:     0.4.8-0               conda-forge          
    proj4:            4.9.3-5               conda-forge          
    prompt_toolkit:   1.0.15-py36_0         conda-forge          
    psutil:           5.4.3-py36_0          conda-forge          
    ptyprocess:       0.5.2-py36_0          conda-forge          
    pygments:         2.2.0-py36_0          conda-forge          
    pynio:            1.5.1.dev0-py36_1     conda-forge/label/dev
    pyparsing:        2.2.0-py36_0          conda-forge          
    python:           3.6.5-0               conda-forge          
    python-dateutil:  2.7.2-py_0            conda-forge          
    pytz:             2018.3-py_0           conda-forge          
    pyyaml:           3.12-py36_1           conda-forge          
    pyzmq:            17.0.0-py36_4         conda-forge          
    readline:         7.0-0                 conda-forge          
    scipy:            1.0.1-py36hfc37229_0  defaults             
    setuptools:       39.0.1-py36_0         conda-forge          
    simplegeneric:    0.8.1-py36_0          conda-forge          
    six:              1.11.0-py36_1         conda-forge          
    sortedcontainers: 1.5.9-py36_0          conda-forge          
    sqlite:           3.20.1-2              conda-forge          
    tblib:            1.3.2-py36_0          conda-forge          
    tk:               8.6.7-0               conda-forge          
    toolz:            0.9.0-py_0            conda-forge          
    tornado:          5.0.1-py36_1          conda-forge          
    traitlets:        4.3.2-py36_0          conda-forge          
    util-linux:       2.21-0                defaults             
    wcwidth:          0.1.7-py36_0          conda-forge          
    wheel:            0.31.0-py36_0         conda-forge          
    xarray:           0.10.2-py36_0         conda-forge          
    xerces-c:         3.2.0-0               conda-forge          
    xz:               5.2.3-0               conda-forge          
    yaml:             0.1.7-0               conda-forge          
    zeromq:           4.2.5-1               conda-forge          
    zict:             0.1.3-py_0            conda-forge          
    zlib:             1.2.11-0              conda-forge          

Proceed ([y]/n)?
darothen-cc commented 6 years ago

Oddly, no. I spun up a fresh VM, grabbed the latest Miniconda3-latest-Linux-x86_64.sh, and executed exactly the same command. The vanilla environment just contends that pynio and python=3.6 are in conflict.

ocefpaf commented 6 years ago

Can you post the contents of your .condarc? Here is mine BTW:

> cat ~/.condarc
show_channel_urls: true
add_pip_as_python_dependency: true
channels:
  - conda-forge
  - defaults
create_default_packages:
  - pip
  - ipykernel
darothen-cc commented 6 years ago

My .condarc was completely empty. However, I copied/pasted yours and it miraculously worked...

ocefpaf commented 6 years ago

Probably b/c the defaults added pynio to the main channel and conda was trying to get that for you. The the .condarc conda will look on conda-forge first.

Note that, according to the docs, the --channel should act like that .condarc configuration and put conda-forge on top of the search, but that never worked properly for me and I have no idea if that works for sub-channels (labels).