The future of NCL and the pivot to Python

[marylhaley]

This GitHub issue was created to give NCL users and the earth science community at large a chance to comment on the NCAR decision to adopt Python as the scripting language platform of choice for future development of analysis and visualization tools.

Please read this open letter to NCL users to understand what kind of impact this will have on the future of NCL.

We have added an FAQ to answer some of the common user questions we received after the announcement.

[Peter-Gibson]

This saddens me - but is understandable. One thing I truly enjoy about NCL is the online documentation and its layout. What percentage of NCL functions will make their was into PyNGL - and over what timeframe?

[marylhaley]

@Peter-Gibson the plan is to determine which NCL computational routines are not already available in Python that are popular with users, and which NCL computational routines are not worth implementing either because they are already available in another Python module or they are no longer relevant. We are not ready to quote percentages right now as we are still new to this particular effort.

We are currently wrapping a suite of NCL computational routines into Python that are part of the Climate Diagnostics Variability Package, developed by Adam Phillips (one of the NCL Advisory Panel members). This package is a nice 'real-world' application of NCL which helps us identify some of the more popular routines first. We've identified about 40 functions in the CVDP that need to be wrapped into Python, while the remaining 80 to 90 ones we've identified as being already implemented in other Python packages, like numpy and xarray.

This 'CVDP Pilot Project', as we call it, is currently an internal GitHub project right now, but the scheduled timeframe for its first public release is later this summer. Here is a roadmap that provides more information. Note that these new routines will not be going into PyNGL, but rather in a separate not-yet-named Python module ("PyNCOMP").

After this point, we will look to our user base and our own example suite to help determine further functions to wrap.

We would love to hear from users who do plan to "pivot to Python" what NCL computational routines are the most critical for them. Communicating this information through this GitHub issue, or by submitting a new GitHub issue is an excellent way to vote on your favorite functions.

[dongli]

NCL provides a one-stop environment for processing and visualizing data, especially the classified resources which we have gotten familiar with for years :thumbsup:. I hope these resources can be mapped to Python package equally (I have never gotten comfortable with Matplotlib :joy:). Meanwhile, I hope we could only need to install one Python package instead a bunch of packages. Anyway, thanks for providing the horsepower for us!

[Try2Code]

Dear NCL developers - thx for taking this important decision. It must have been a hard task to do with all the users and code heritage. I am looking forward to this because (1) I like the plotting features of NCL and (2) had the impression that NCL is somewhat limited in terms of language features (use of command-line arguments, user-defined data structures, key-value stores).

Python is certainly the way to go IMO from its reputation and wide-spread use in the field. But when it comes to python libraries, there area some lines I need to drop here: Please stick some of your concepts of how to describe a plot - don't let matplotlib do every thing, since it has some awkward features. In combination with Basemap, Cartopy or whatever related libraries this leads to strange patterns. NCL is the opposite: you get all from a single vendor (which I consider a good thing). I just want to encourage you to stick to your concepts. I'd love to see a coherent approach for plotting climate and weather data esp. in combination with so powerful libraries like XArray and Dask.

Big thx for your support of the community!

[Peter-Gibson]

Hi Mary,

Thanks for the support and work on NCL over the years - its an amazing resource. I talked with other NCL users in our research group (who also use Python) and a few thoughts we collectively had:

1. NCL's map plotting produces nicer plots with less effort and with the resources all in one place. This functionality into Python would be hugely valuable. I can make maps in Python look NCL-like but with many many more lines of code needed.
2. NCL's functions for computing gradients (both using cfd and spherical harmonics) and NCL's functions for divergence, divergent wind components, vorticity, sf, vp are great because these are non-trivial to calculate - I don't think a full suite of these exist in Python - at least not all in one place.
3. It would be great if NCL's website stayed online into the future (even if it wasn't updated). This is an highly valuable one-stop-shop resource for functions/examples/code.

Thanks again!

[wblumberg]

Regarding thought number 2: windspharm (https://ajdawson.github.io/windspharm/latest/) might be what you're looking for. MetPy (https://unidata.github.io/MetPy/latest/index.html) has some pretty nice functions for calculating those too; they use finite differencing.

[billboos]

One thing that has been nice about NCL is the ability to install it and go -- setting up the python environment can be a nightmare for new users, especially if you are trying to do so in an HPC setting. If, as part of this pivot, NCAR developers could provide some instructions, guides, or package management to help atmospheric scientists learning to use python in an HPC environment, that would be a major help. I switched from Matlab & NCL to python about a year ago, and I almost gave up in frustration after trying to get my Anaconda environment to work as a jupyter hub kernel on the clusters I use.

[Try2Code]

maybe NCL could be shipped with a conda environment file to make this easier. It can be a nightmare to get libs to their correct versions ...

One thing that has been nice about NCL is the ability to install it and go -- setting up the python environment can be a nightmare for new users, especially if you are trying to do so in an HPC setting. If, as part of this pivot, NCAR developers could provide some instructions, guides, or package management to help atmospheric scientists learning to use python in an HPC environment, that would be a major help. I switched from Matlab & NCL to python about a year ago, and I almost gave up in frustration after trying to get my Anaconda environment to work as a jupyter hub kernel on the clusters I use.

[dongli]

maybe NCL could be shipped with a conda environment file to make this easier. It can be a nightmare to get libs to their correct versions ...

And it would be nicer if we could install PyNGL through the standard pip.

[mvarma04]

I also see that scientific community is moving towards python. If anyone wants to take a transition from NCL to Python, I think it would be a great help for the NCL users to have some examples particularly relating to different types of plots. For the plotting purpose, I am totally depending on NCL right now. So, pages like this (http://www.ncl.ucar.edu/Applications/NCL_to_Python/) would be of great help.

[BarryHLynn]

I just want to mention that I believe that the management and developers are quite underestimating the utility and potential of NCL. Almost every time I need to do something, I find a suitable example. Moreover, it is an incredibly logical language, often built on incredibly useful and simple to use commands (like conform). I am, at time, imply amazed at the care that went into building the logic of NCL and how the parts actually blend together to work seamlessly. NCL should not be abandoned -- it has a large, eager, and helpful user base with years of experience, willing to lend their ideas towards future development, while helping to maintain the integrity of the code. What's in Python that's not in NCL? There are plenty using Python -- let them use it and let them suffer from its inconsistencies. In the meantime, just build NCL where it's needed, taking the best ideas in Python -- but don't abandon a system that is and will be fundamentally better for its users (in the atmospheric sciences) -- for years to come.

[BarryHLynn]

Hello:

I just wanted to make sure that the developers are aware of critical NCL functions (at least from my point of view).

For example, 1) the calendar functions, inverse and regular. 2) The ability to subset strings and obtain dates for example from the strings. 3) The ability to use attributes to obtain times (e.g., from the GFS data). 4) The ability to call shell commands from within NCL. 5) The ability to parse variables in a list, e.g., CSV files read as single lines and then parsed into separate variables. 6) The ability to "WRAPIT" Fortran programs. 7) The ability to subset arrays and conform them.
8) To be able to find min/max, and convert from 2d to 1d arrays for doing so.

Perhaps the biggest obstacle is learning how to do all of these things in Python, let alone making maps, overlays, panels.

It seems to me that it will take a while to create Python equivalent examples as exist online with explicit detail on the NCL pages.

[harrytheharris]

Thanks to all NCL devs, past and present, for a fantastic product: reliable, full-featured, and with a bottomless pool of assistance no matter your difficulty. I'm sure this decision wasn't taken lightly; I'd been expecting something like this for some years and I'm glad to find my worst fears unrealised - that the existing implementation will survive, at least for now. Having recently been obliged to work in R, as a result of inheriting someone else's work, I now view the challenge of new languages with trepidation; however, this gives me an excuse to play with Python without pressure.

[brianmapes]

One aspect of the transition might be to notice that this exists: https://github.com/suvarchal/IPyNCL

Good luck to all users and we welcome your comments!

[apendergrass]

Hi all,

I was just processing some data in NCL, and while it's running, I wanted to check in and see whether the functions I'm using will be supported. I rely on NCL heavily for atmospheric-science and earth-specific functionality. I get the sense from the transition materials are more visualization heavy.

Right now I'm calculating divergence on the CESM gaussian grid using uv2dvG. Yesterday I was doing vertical interpolation with vinth2p_ecmwf and also vibeta. The thing I probably use NCL for most is regridding, especially ESMF_regrid. Are these or will these be available in PyNCL?

Thanks, Angie

[Neeldip]

Hi, I was just gonna start learning NCL untill I saw the planned transition to python. I am in confusion now whether I should start now with the non Python ncl. I want suggestions. Could someone give an idea of when the new user guide/documentation will be available? Regards, Neeldip

[clyne]

Hi Neeldip,

I'm not sure that I'll be able to answer your question satisfactorily, but let me share a couple of pieces of information that might help you make your decision.

The GeoCAT team (formerly NCL team) are committed to developing tools for the Python ecosystem, and will no longer be adding new features to NCL. However, NCL is mature and powerful, and will continue to be supported for the foreseeable future. Furthermore, the GeoCAT effort is in the early stages and very little has been released at this point. You can find up-to-date information on the GeoCAT project from the new website: https://geocat.ucar.edu. Of particular interest may be the roadmap available here: https://github.com/NCAR/geocat-comp/projects

Hope this helps.

[niweicheng]

Hi, I am wondering if all functions in NCL will be transformed into python. I would like to calculate the stream function and velocity potential with the function 'uv2sfvpf'. However, I cannot find similar functions in Geocat repository.

[clyne]

Hi, I am wondering if all functions in NCL will be transformed into python. I would like to calculate the stream function and velocity potential with the function 'uv2sfvpf'. However, I cannot find similar functions in Geocat repository.

The triaging of NCL functions for inclusion in GeoCAT is complex. Functions will only be added if there is a need and an existing alternative does not already exist. I suggest you open an enhancement request for the function you'd like to see ported on the GeoCAT-comp repo: https://github.com/NCAR/geocat-comp/issues

[billboos]

All: if you’re interested in python functions for streamfunction, velocity potential, etc, you should consider the windspharm package:

https://ajdawson.github.io/windspharm/latest/ https://ajdawson.github.io/windspharm/latest/

My vague understanding is that windspharm uses the python-wrapped spherepack, and my group has found these to be faster than the NCL routines and to have more functionality (e.g. you can truncate calculations at a chosen wavenumber, etc). My default “import” stack now includes:

import windspharm import spharm import _spherepack

Cheers, Bill

On Dec 15, 2020, at 3:58 PM, clyne notifications@github.com wrote:

Hi, I am wondering if all functions in NCL will be transformed into python. I would like to calculate the stream function and velocity potential with the function 'uv2sfvpf'. However, I cannot find similar functions in Geocat repository.

The triaging of NCL functions for inclusion in GeoCAT is complex. Functions will only be added if there is a need and an existing alternative does not already exist. I suggest you open an enhancement request for the function you'd like to see ported on the GeoCAT-comp repo: https://github.com/NCAR/geocat-comp/issues https://github.com/NCAR/geocat-comp/issues — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCAR/ncl/issues/64#issuecomment-745641850, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBYEIJL34EOJPTU2UCKWVDSU7Z2LANCNFSM4GUQ7SIQ.

On Dec 15, 2020, at 11:16 AM, niweicheng notifications@github.com wrote:

Hi, I am wondering if all functions in NCL will be transformed into python. I would like to calculate the stream function and velocity potential with the function 'uv2sfvpf'. However, I cannot find similar functions in Geocat repository.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCAR/ncl/issues/64#issuecomment-745507422, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBYEIPKNFW6MXOE5DRS4FLSU6YYZANCNFSM4GUQ7SIQ.

On Apr 20, 2019, at 8:54 AM, Angeline G Pendergrass notifications@github.com wrote:

Hi all,

I was just processing some data in NCL, and while it's running, I wanted to check in and see whether the functions I'm using will be supported. I rely on NCL heavily for atmospheric-science and earth-specific functionality. I get the sense from the transition materials are more visualization heavy.

Right now I'm calculating divergence on the CESM gaussian grid using uv2dvG https://www.ncl.ucar.edu/Document/Functions/Built-in/uv2dvG-1.shtml. Yesterday I was doing vertical interpolation with vinth2p_ecmwf https://www.ncl.ucar.edu/Document/Functions/Built-in/vinth2p_ecmwf.shtml and also vibeta https://www.ncl.ucar.edu/Document/Functions/Built-in/vibeta.shtml. The thing I probably use NCL for most is regridding, especially ESMF_regrid https://www.ncl.ucar.edu/Document/Functions/ESMF/ESMF_regrid.shtml. Are these or will these be available in PyNCL?

Thanks, Angie

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCAR/ncl/issues/64#issuecomment-485137952, or mute the thread https://github.com/notifications/unsubscribe-auth/AGBYEIOYJIXZS33YKLUVZZLPRM4FHANCNFSM4GUQ7SIQ.