Closed darothen closed 6 years ago
Any progress on Py3.x support. Xarray is using Pynio as a ingest for grib... I am nearly 100% 3.x now and don't want to go back.
Hi Scott and Daniel, We are currently working on Python 3 support for PyNIO as a high priority task. The developer is currently optimistic that it won't take too long so hopefully it will be available fairly soon. -dave
On Mon, Jun 26, 2017 at 9:23 AM, Scott notifications@github.com wrote:
Any progress on Py3.x support. Xarray is using Pynio as a ingest for grib... I am nearly 100% 3.x now and don't want to go back.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NCAR/pynio/issues/10#issuecomment-311092301, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbWwlwJPMJwnQhDZtEgpuUNpkV797ks5sH8zlgaJpZM4LXoMx .
I'm working on an ML project for NASA (related to @darothen 's mention of earthio
above). We're glad to hear Python 3 support is coming for pynio
. Thanks!
@david-ian-brown any late summer updates on this? I have an application using HRRR GRIB2 output which leans heavily on xarray; if I could skip the step of converting GRIB2 to NetCDF (which would be possible with PyNIO support for Python 3), it would greatly help.
I'm also super excited to see pynio
with python 3 support.
But for anyone reading this thread: in the meantime, if you use anaconda, you can easily create a python 2.7 virtual environment. This link has some great info. Create the environment with conda create -n python2 python=2.7 anaconda
(where python2
is the name I chose for the virtualenv), install the relevant packages with conda install -n python2 xarray
and conda install -n python2 -c conda-forge pynio
, activate the environment from the command-line with source activate python2
or . activate python2
, and launch an ipython terminal, jupyter notebook, etc while the environment is "activated."
This is useful for any other packages that haven't yet been ported to python 3.
@lukelbd that's true - but the scientific Python community is quickly moving away from Python 2, and there is little reason to maintain two fully functioning development environments for daily work. Virtual environments aren't a solution here in many cases. For instance, I'm building a workflow to ingest/parse GRIB2 output via xarray for work, and our entire stack is Python 3; it's hugely disruptive to maintain a separate Python stack on all our cloud services just to read those files into our system.
@darothen you're absolutely right; just wanted to share a band-aid solution (viable for not-too-intensive workflows) until the python 3 update arrives. Though it looks like that will be imminent, so maybe it was not necessary.
Unfortunately, the PyNIO port is taking a little longer than I expected, so I wouldn't say that release is imminent. It's currently in the debug stage, soon approaching testing.
Thanks for hte update @bladwig1 . Is there anything the community to do at this stage to help out?
PyNIO for Python 3.x is now merged in to the 'develop' branch on GitHub. However, an official release on conda-forge won't take place until a large amount of bug fixes related to compound data types, HDF5, etc are included. That being said, you should be able to pull down PyNIO from GitHub and start trying it out with Python 3.x.
There is now a "dev" snapshot build on conda (ncar channel), which you can get by doing:
conda create -n (your_env_name) -c ncar -c conda-forge pynio=dev python=3
If you notice any issues, please create a new GitHub issue.
Thanks!
Hey all - I know this thread is technically closed, but there's an issue with this old protocol.
Somewhere along the line, a dependency issue was introduced. If you install a clean Python 3 environment with just xarray, a la:
$ conda create -n test_nio -c conda-forge python=3.6 xarray
then activate it and attempt to install pynio as referenced above:
$ conda install -c ncar pynio=dev
you'll run into an issue with libnetcdf versions, where it requests a downgrade:
The following packages will be DOWNGRADED:
libnetcdf: 4.5.0-3 conda-forge --> 4.4.1.1-10 conda-forge
netcdf4: 1.3.1-py36_2 conda-forge --> 1.3.1-py36_1 conda-forge
Somewhere along the line, hdf5
gets messed up and it renders pynio un-importable. On my MacBook I'll get import errors such as:
ImportError: dlopen(/Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libhdf5_hl.10.dylib
Referenced from: /Users/daniel/anaconda/envs/test_nio/lib/python3.6/site-packages/PyNIO/_nio.cpython-36m-darwin.so
Reason: image not found
Any thoughts and/or solutions? This is new behavior; last time I helped someone get set-up with pynio/Python 3 back in January we did not run into this problem.
Yeah add me to the list of people this no longer works for..
The issue here is a mismatch between the ncar
channel and the conda-forge
channel for the dependencies used ni both pynio
and xarray
. I recommend to release the dev version of pynio
in a sub-label in conda-forge
, that way people could do:
$ conda install --channel conda-forge/label/dev pynio
this would be in sync with the other dependencies we have in conda-forge
and would avoid this issue.
Thanks for reporting this @darothen and @scollis, I'm looking into this now.
@ocefpaf, I will create an issue to further discuss your idea on the pynio-feedstock repo.
I can confirm that @ocefpaf's solution has worked just fine, and fixes the dependency issues.
Thanks to @khallock who promptly created the dev version on conda-forge
.
PS: he also created a ncl
dev package too, install it with
conda install --channel conda-forge/label/dev ncl
:tada:
Thanks all!
-sent from a mobile device-
On Mar 18, 2018, at 1:57 PM, Filipe notifications@github.com wrote:
Thanks to @khallock who promptly created the dev version on conda-forge.
PS: he also created a ncl dev package too, install it with
conda install --channel conda-forge/label/dev ncl 🎉
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
FWIW, there may be a new issue - the resolved method isn't working, and is forcing a downgrade to Python 2.7. I think I've traced this down to an older version of cyordereddict which is pinned to Python 2.7, and is a dependency of xarray.
@darothen what happens if you specify the python version too? Like
conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio
The conda solver tries to wiggle its way out of complex dependencies by downgrading package version and build number, sometimes it even breaks channel preference! So the more you specify in the install command the better.
@ocefpaf No dice, unfortunately:
Works for me, do you have other channel in your .condarc
?
> conda create --name TEST --channel conda-forge/label/dev python=3.6 xarray pynio
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.4.10
latest version: 4.5.0
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: /home/filipe/miniconda3/envs/TEST
added / updated specs:
- ipykernel
- pip
- pynio
- python=3.6
- xarray
The following packages will be downloaded:
package | build
---------------------------|-----------------
krb5-1.14.6 | 0 4.0 MB conda-forge
backcall-0.1.0 | py_0 13 KB conda-forge
distributed-1.21.5 | py36_0 772 KB conda-forge
bokeh-0.12.15 | py36_0 4.2 MB conda-forge
pynio-1.5.1.dev0 | py36_1 4.3 MB conda-forge/label/dev
wheel-0.31.0 | py36_0 62 KB conda-forge
msgpack-python-0.5.6 | py36_0 300 KB conda-forge
ipython-6.3.0 | py36_0 1.0 MB conda-forge
------------------------------------------------------------
Total: 14.7 MB
The following NEW packages will be INSTALLED:
backcall: 0.1.0-py_0 conda-forge
bokeh: 0.12.15-py36_0 conda-forge
boost: 1.66.0-py36_1 conda-forge
boost-cpp: 1.66.0-1 conda-forge
bottleneck: 1.2.1-py36_1 conda-forge
bzip2: 1.0.6-1 conda-forge
ca-certificates: 2018.1.18-0 conda-forge
cairo: 1.14.10-0 conda-forge
certifi: 2018.1.18-py36_0 conda-forge
click: 6.7-py_1 conda-forge
cloudpickle: 0.5.2-py_0 conda-forge
curl: 7.59.0-0 conda-forge
cytoolz: 0.9.0.1-py36_0 conda-forge
dask: 0.17.2-py_0 conda-forge
dask-core: 0.17.2-py_0 conda-forge
decorator: 4.2.1-py36_0 conda-forge
distributed: 1.21.5-py36_0 conda-forge
expat: 2.2.5-0 conda-forge
fontconfig: 2.12.6-0 conda-forge
freetype: 2.8.1-0 conda-forge
freexl: 1.0.5-0 conda-forge
g2clib: 1.6.0-5 conda-forge
geos: 3.6.2-1 conda-forge
geotiff: 1.4.2-1 conda-forge
gettext: 0.19.8.1-0 conda-forge
giflib: 5.1.4-0 conda-forge
glib: 2.55.0-0 conda-forge
h5netcdf: 0.5.0-py_0 conda-forge
h5py: 2.7.1-py36_2 conda-forge
hdf4: 4.2.13-0 conda-forge
hdf5: 1.10.1-2 conda-forge
hdfeos2: 2.19.1-8 conda-forge
hdfeos5: 5.1.16-0 conda-forge
heapdict: 1.0.0-py36_0 conda-forge
icu: 58.2-0 conda-forge
intel-openmp: 2018.0.0-8 defaults
ipykernel: 4.8.2-py36_0 conda-forge
ipython: 6.3.0-py36_0 conda-forge
ipython_genutils: 0.2.0-py36_0 conda-forge
jasper: 1.900.1-4 conda-forge
jedi: 0.11.1-py36_0 conda-forge
jinja2: 2.10-py36_0 conda-forge
jpeg: 9b-2 conda-forge
json-c: 0.12.1-0 conda-forge
jupyter_client: 5.2.3-py36_0 conda-forge
jupyter_core: 4.4.0-py_0 conda-forge
kealib: 1.4.7-4 conda-forge
krb5: 1.14.6-0 conda-forge
libdap4: 3.18.3-2 conda-forge
libffi: 3.2.1-3 conda-forge
libgcc-ng: 7.2.0-hdf63c60_3 defaults
libgdal: 2.2.4-0 conda-forge
libgfortran: 3.0.0-1 defaults
libgfortran-ng: 7.2.0-hdf63c60_3 defaults
libiconv: 1.15-0 conda-forge
libkml: 1.3.0-6 conda-forge
libnetcdf: 4.4.1.1-10 conda-forge
libpng: 1.6.34-0 conda-forge
libpq: 9.6.3-0 conda-forge
libsodium: 1.0.16-0 conda-forge
libspatialite: 4.3.0a-19 conda-forge
libssh2: 1.8.0-2 conda-forge
libstdcxx-ng: 7.2.0-hdf63c60_3 defaults
libtiff: 4.0.9-0 conda-forge
libxml2: 2.9.8-0 conda-forge
locket: 0.2.0-py36_1 conda-forge
markupsafe: 1.0-py36_0 conda-forge
mkl: 2018.0.2-1 defaults
mkl_fft: 1.0.1-py36_1 conda-forge
mkl_random: 1.0.1-py36_0 conda-forge
msgpack-python: 0.5.6-py36_0 conda-forge
ncurses: 5.9-10 conda-forge
netcdf4: 1.3.1-py36_1 conda-forge
numpy: 1.14.2-py36hdbf6ddf_1 defaults
openjpeg: 2.3.0-2 conda-forge
openssl: 1.0.2n-0 conda-forge
packaging: 17.1-py_0 conda-forge
pandas: 0.22.0-py36_0 conda-forge
parso: 0.1.1-py_0 conda-forge
partd: 0.3.8-py36_0 conda-forge
pcre: 8.41-1 conda-forge
pexpect: 4.4.0-py36_0 conda-forge
pickleshare: 0.7.4-py36_0 conda-forge
pip: 9.0.3-py36_0 conda-forge
pixman: 0.34.0-1 conda-forge
poppler: 0.61.1-3 conda-forge
poppler-data: 0.4.8-0 conda-forge
proj4: 4.9.3-5 conda-forge
prompt_toolkit: 1.0.15-py36_0 conda-forge
psutil: 5.4.3-py36_0 conda-forge
ptyprocess: 0.5.2-py36_0 conda-forge
pygments: 2.2.0-py36_0 conda-forge
pynio: 1.5.1.dev0-py36_1 conda-forge/label/dev
pyparsing: 2.2.0-py36_0 conda-forge
python: 3.6.5-0 conda-forge
python-dateutil: 2.7.2-py_0 conda-forge
pytz: 2018.3-py_0 conda-forge
pyyaml: 3.12-py36_1 conda-forge
pyzmq: 17.0.0-py36_4 conda-forge
readline: 7.0-0 conda-forge
scipy: 1.0.1-py36hfc37229_0 defaults
setuptools: 39.0.1-py36_0 conda-forge
simplegeneric: 0.8.1-py36_0 conda-forge
six: 1.11.0-py36_1 conda-forge
sortedcontainers: 1.5.9-py36_0 conda-forge
sqlite: 3.20.1-2 conda-forge
tblib: 1.3.2-py36_0 conda-forge
tk: 8.6.7-0 conda-forge
toolz: 0.9.0-py_0 conda-forge
tornado: 5.0.1-py36_1 conda-forge
traitlets: 4.3.2-py36_0 conda-forge
util-linux: 2.21-0 defaults
wcwidth: 0.1.7-py36_0 conda-forge
wheel: 0.31.0-py36_0 conda-forge
xarray: 0.10.2-py36_0 conda-forge
xerces-c: 3.2.0-0 conda-forge
xz: 5.2.3-0 conda-forge
yaml: 0.1.7-0 conda-forge
zeromq: 4.2.5-1 conda-forge
zict: 0.1.3-py_0 conda-forge
zlib: 1.2.11-0 conda-forge
Proceed ([y]/n)?
Oddly, no. I spun up a fresh VM, grabbed the latest Miniconda3-latest-Linux-x86_64.sh
, and executed exactly the same command. The vanilla environment just contends that pynio and python=3.6 are in conflict.
Can you post the contents of your .condarc
? Here is mine BTW:
> cat ~/.condarc
show_channel_urls: true
add_pip_as_python_dependency: true
channels:
- conda-forge
- defaults
create_default_packages:
- pip
- ipykernel
My .condarc
was completely empty. However, I copied/pasted yours and it miraculously worked...
Probably b/c the defaults added pynio
to the main channel and conda was trying to get that for you. The the .condarc
conda will look on conda-forge first.
Note that, according to the docs, the --channel
should act like that .condarc
configuration and put conda-forge on top of the search, but that never worked properly for me and I have no idea if that works for sub-channels (labels).
In support of upstream Python 3 compatibility (NCAR/PyReshaper#26).