search

conda-forge / python-eccodes-feedstock

A conda-smithy repository for python-eccodes.
BSD 3-Clause "New" or "Revised" License
6 stars 16 forks source link

python3k bindings #3

Closed kmuehlbauer closed 5 years ago

kmuehlbauer commented 8 years ago

This extends to the discussion we had in this PR proposing py3k patch, which is already merged.

For the first there's already an answer. ECMWF asked if the patch could be made to cover both python versions ("all versions" was their exact term). This is good news I think, that they are willing to accept those patches.

For the second there is no answer yet. We could start from this repo here, where I imported the eccodes-0.13.0-sources and tested the python3k-patch with TravisCi on linux. To get things along with ECMWF we could combine efforts and make the patch more generic to have it work for all python versions. @QuLogic already made some comments in this issue). There is also GribApi.XP which is a friendly fork of the ECMWF gribapi. Therefore pinging @0x1mason, if he has enough interest to have his gribapi experience within the successor eccodes. Anyway, since gribapi and eccodes share many things (from the build system to the python bindings), we can share things.

Pinging @pelson, @ocefpaf, the @conda-forge/python-ecmwf_grib people and @QuLogic for getting their opinions on how to further proceed. If there are more people interested in this I encourage everybody to invite them to this discussion.

Hoping to get python3k support in eccodes with the help of the community.

kmuehlbauer commented 8 years ago

There is response from ECMWF regarding the second issue above about community collaboration.

We are definitely looking forward to any kind of collaboration regarding our software packages. In principle all that you suggest should be fine but I will double check the development team. What we could also look at is the option of giving you access to a fork of the ecCodes repository in our own environment (Bitbucket Server) so that you could push changes and make pull requests with the option of integrating any changes upstream if approved.

My answer:

this is really great news. As I have also a bitbucket account this would help very much. At least we easily could see, if the changes to be added by the community are colliding with changes in your dev-version. This also seems the fastest way to get changes into your environment. Anyway, we would need to base our work on your latest release version. But as already said, having the opportunity to closely collaborate via bitbucket will tremendously speed up things. Thanks for taking the time. Waiting for your response from dev-team.

This is all in all very promising. Do you already have ideas on how to glue the different parts together?

ocefpaf commented 8 years ago

Awesome news @kmuehlbauer! :tada:

kmuehlbauer commented 7 years ago

Unfortunately I didn't had any time to pursue this. The eccodes devs released version 2.0.0 which is just rolled out in conda-forge. Due to the changes in the code, the py3k-patch will not work out of the box, meaning the 2.0.0 is just python 2.7 for now.

ocefpaf commented 7 years ago

No problem. That is not a simple task! And quite time consuming.

kynan commented 7 years ago

@kmuehlbauer picked this up again in #18

kmuehlbauer commented 7 years ago

@kynan @shahramn @erget IIRC the main problem regarding py3k was the file-object handling. Has this changed within the new version (2.X). The patch was a more or less crude hack then, to fix py3k for the file-handling. Can someone elaborate on eccodes current file-handling?

kynan commented 7 years ago

@kmuehlbauer This is not yet in ecCodes 2.X. However there are new bindings in the works that support both Python 2 and 3. No ETA yet when those will be available - but things are moving...

kmuehlbauer commented 7 years ago

However there are new bindings in the works that support both Python 2 and 3

OK, this is very promising. Unfortunately I didn't had time to keep the contact with ECMWF (see top-post).

kynan commented 7 years ago

@kmuehlbauer No worries! The door is open, contact @shahramn if you'd like to help out :)

four43 commented 6 years ago

It seems like the ecCodes folks have Python 3.x support. Debian has a python3 package for this. Is this something that could be pursued again?

I'd love to help but I'm having a heck of a time getting it compiled and installed and working in a python3 environment.

knedlsepp commented 6 years ago

This is still not officially supported upstream. Although this probably won't be of much help, since the issue tracker is not open to the general public, this is the ticket that tracks python3 support https://software.ecmwf.int/issues/browse/ECC-264.

four43 commented 6 years ago

Thanks for the quick response on this, glad we can kind of keep the conversation rolling at least. I appreciate it.

akrherz commented 6 years ago

@knedlsepp could I trouble you for an update on that ECC-264 Jira ticket you referenced, since I can't see it :)

kmuehlbauer commented 5 years ago

@StephanSiemen Can you say something about the status of python3 bindings? Since many major packages already abandoned python2, this is getting problematic.

StephanSiemen commented 5 years ago

@kmuehlbauer, we are aware of users waiting for this. Just today we worked on it. But moving to Python3 turned out to be much harder than expected. The file I/O in ecCodes is modelled on the C/Fortran interfaces and relied on the FILE* pointer to hold the state of the interaction with the GRIB file. This low-level file access is not supported anymore in Python3 with the new I/O interfaces. So far we aimed to keep the interface as unchanged as possible, but we doubt we can achieve this. If there is anyone out there knowledgeable on SWIG and Python low level file I/O give us a shout! ;-) You might have heard that we at ECMCF now setting on the (higher-level Python3-only) cfgrib (https://pypi.org/project/cfgrib/) interface for accessing GRIB files using ecCodes, but this will not stop us from trying to support the low-level legacy API, as we know that there are many users of it. Sorry, for not having any better news :-( We will try to keep the issue up-to-date with any developments ...

kmuehlbauer commented 5 years ago

Thanks @StephanSiemen, for the insight and the pointer to cfgrib.

erget commented 5 years ago

Hi @StephanSiemen is there any news concerning Python3 and BUFR? cfgrib is great but as I understand it it only covers GRIB. And of course it would be useful to be able to write as well as read using whatever solution ECMWF follows.

kmuehlbauer commented 5 years ago

@four43 Sorry, your debian hint somehow got lost here. After reading the whole thread again I just had a quick look into the debian patch (https://sources.debian.org/patches/eccodes/2.9.2-2/). There are several patches debian does to eccodes to get python3 compatibility.

But does anyone have actual experience this is really working?

From my perspective, it should be possible to apply the relevant patches here and give this a try. @StephanSiemen do you have had any contact with debian on this?

StephanSiemen commented 5 years ago

@kmuehlbauer, @four43, yes we are aware of the Debian/ubuntu patches for a while. They are addressing issues only far enough to allow the importing of these packages packages under Python 3 without conflict. When you actually use the Python3 packages you get errors related to access the file itself. This is the issue we are currently fighting with. The file I/O was updated to the Python 3 one, but it is not equivalent and fails when you open or write a GRIB file (the purpose of the package after all).

StephanSiemen commented 5 years ago

@erget, yes we are in the early stages of building also an equivalent high-level BUFR interface. It's based on what we do in Metview: Users specify filters on the BUFR messages and get returned a pandas array with the selected information. Keep you posted on this ...

StephanSiemen commented 5 years ago

Please note we released ecCodes 2.10 today which includes a beta version of a Python 3 interface. I already fired off a pull request at https://github.com/conda-forge/python-eccodes-feedstock/pull/41 ... let's see how it goes :-)

Update: PR

kmuehlbauer commented 5 years ago

The packages were built for Python 2.7, 3.6 and 3.7 and should be available shortly.

We can leave this issue open for the time being and report anything related here.

kmuehlbauer commented 5 years ago

Tested reading bufr files using Python 3.6. Works smooth! Huge thanks to all at ECMWF for making this happen.

BTW, does someone have recent local DWD bufr tables? The ones on opendata.dwd.de are outdated.

ocefpaf commented 5 years ago

We are building it for Python 3 now, right? Closing this but please re-open if I got it wrong.