Open mattiarighi opened 5 years ago
bouweandela
commented
5 years ago external diagnostic packages that are (or are going to be) implemented in the ESMValTool and are written in NCL still exist (e.g., CVDP).
A bit of good news here is that they are translating this particular example to Python as a pilot, see: https://github.com/NCAR/ncl/issues/64#issuecomment-461256463
bouweandela
commented
5 years ago From the roadmap it also looks like there will be training available.
zklaus
commented
5 years ago Thanks for picking this up, @mattiarighi !
stopping NCL support in the ESMValTool right now is not realistic and the upcoming v2.0 will still need to support NCL (for the diagnostics and the CMORizers).
I completely agree that this is a process, cannot happen over night, and should have (almost) zero bearing on v2.0 (the exception just being that, all else being equal, I wouldn't prefer NCL for new developments). I think your suggestion below for recommendations to developers is a good thing, more on that below.
external diagnostic packages that are (or are going to be) implemented in the ESMValTool and are written in NCL still exist (e.g., CVDP).
@bouweandela has that covered. I think therefore CVDP isn't a blocker, but maybe we should collect all such packages that we have in some place (extra ticket, wiki?)?
several (how many?) ESMValTool developers can only contribute NCL diagnostics and may not want to contribute code in another language or may not have the resources to do that. A survey across all ESMValTool users and developers might help us to get a better feeling on this.
That is unfortunate, but adding substantial new NCL code seems to go in the wrong direction. In most cases those developers will have to adapt to the changing environment for reasons entirely outside of ESMValTool.
according to NCAR: "we want to stress that NCL is not going away. NCL users will be able to download NCL and execute their scripts for the foreseeable future". It would be good to get in touch with the NCL development team to better understand the actual perspectives. Are we talking about months or years? What happens in case of NCL installation issues on new systems? Would support still be provided in such cases? Etc.
:+1: Though of course, as you know, intentions aside, the limits will always be funding and man-power. Usually, few people with the necessary skill are eager to maintain things after a few years without the mandate to change them.
after the release of v2.0, we should start recommending our NCL develoeprs to switch to Python, maybe using the aforementioned PyNGL package to make this transition smooth (a transition guide by DKRZ is also available).
To formalize this: With the release after v2.0 or with v2.0 itself, we should formally deprecate it. Imho, we can already now give the general recommendation to rather prefer python, making it clear that NCL contributions are still acceptable for now. But if somebody is on the fence on what to choose, they should go for python.
based on my experience in coordinating the v2.0 development, the coding quality standards of the ESMValTool require a non negligible amount of additional resources (for both developers and reviewers), which might discourage the newbies to switch from NCL to python.
Hm, this shouldn't be specific to python. If it seems that the burden on python contributions is higher than on other languages, we should rather see how we can achieve the same high standards in those languages. At the moment we are rather lenient there, probably for lack of automatic qa. As an example, we know that, no matter the language, functions should be short. I find around 45 lines as an upper limit a good guide rule. If in python you make much longer functions, you are likely to get a complexity warning or too-many-local-variables from our code checkers. In NCL this malpractice is still wide-spread.
Bottom line: Quality standards should be equally high for all languages, the effort comparable.
we should be aware that by reducing the multi-language support, we face the risk of losing a (significant?) part of the ESMValTool community.
Interesting question. Looking at the list of contributors (here for the private repository for those who have access), my impression is that most would be comfortable with python. What do you think? Of course, this is just a personal impression. To get a better picture we could do a survey, best limited to those having contributed at least one commit, or try automatic numbercrunching, eg see what fraction of lines added per person where in NCL. I think a survey can be done quickly with (eg) google forms; number crunching would take more effort.
It would be very interesting to hear from NCL developers on this.
valeriupredoi
commented
5 years ago I got 0 NCL lines :rofl:
So this will be easier to discuss when we will have split ESMValTool (at least) into core and diagnostics. The core is and will be Python; the diagnostics are a bit more relaxed and the shepherding of the scientists developing them towards Python (and away from NCL) can be done in a bit more of a relxaed way rather than at gun point if it was for the core
mattiarighi
commented
5 years ago DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.
jvegreg
commented
5 years ago DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.
Just remember that the end of python 2 support seemed to be far far away
valeriupredoi
commented
5 years ago DKRZ is still organizing NCL workshops for the users, so they are probably not planning to retire NCL support soon.
Just remember that the end of python 2 support seemed to be far far away
I am working with Met Office software which is (as expected) Python2 and everytime I try and update Py2 deps I get warnings about the looming dat of January 1st 2020 :grin:
mattiarighi
commented
5 years ago There are some news about the future of NCL:
http://www.ncl.ucar.edu/Document/Pivot_to_Python/september_2019_update.shtml
"To summarize the status of NCL, the project is "feature frozen". CISL does not have plans at this time to add new features to NCL. We will, however, prepare maintenance releases containing critical bug fixes and user-contributed code on an infrequent basis. Thus, there is no reason for members of the community with substantial investments in NCL to consider porting their NCL code to Python (unless you want to take advantage of the improved scalability afforded by our use of Dask)."
bouweandela
commented
3 years ago A new update from 11 November 2020 by the NCL authors:
In January of 2019 NCAR announced plans to transition away from the NCL language and focus efforts on providing geosciences analysis tools that were based on the Python Scientific Ecosystem. Furthermore, we announced that NCL would be put into “maintenance mode”, and due to limited staffing resources, would no longer be actively developed by NCAR.
I wanted to take this opportunity to elaborate further on what “maintenance mode” means for the community, and hopefully relieve anxiety that some may be feeling. Maintenance mode means that NCAR will not add new features to the NCL language or function library, nor will we fix sub-critical software defects. However, NCAR will for as long as we are reasonably able and there is a sufficient demand - put out periodic bug-fix releases and ensure NCL continues to build on currently supported platforms. In addition, we will do our best to timely address installation questions that are related to the Conda package management system. Furthermore, as part of our embracement of Open Development practices, we will facilitate community contributions of all types (e.g. bug fixes, new functions, etc.). Stay tuned for more information on Open Development.
I fully recognize that many in the community have substantial investments in NCL scripts and I do not expect, nor would I advise, translating thousands of lines of NCL code into Python. That said, I would caution against making substantial new investments in workflows based on NCL. To repeat, it is our intent to ensure that existing NCL scripts will continue to run for as long as reasonably possible, but NCAR’s priority for the future will be on modern Python tools.
zklaus
commented
3 years ago NCL in conda-forge has been a bit neglected for a while. This means that it has not been (re-)build against more recent versions of the usual libraries (libgdal, libnetcdf, proj,...) which makes it hard for us to install recent versions that are required by other parts of the ESMValTool eco system.
I have spent a good chunk of the last week and the beginning of this week trying to improve the situation, but this is a real drain on resources and it is not clear how long we can keep this up.
mattiarighi
commented
3 years ago Related to this, you might be interested in the python GeoCAT package, which translates many of the original NCL function in Python.
It could be a way to encourage NCL users to move to Python...
zklaus
commented
2 years ago I think it really is time to phase out NCL. The conda-forge build is broken again and it is clear now that NCAR is no longer performing maintenance. The existing builds will continue to work for a while and we can make some effort to keep the car running, but sooner rather than later this becomes too much of a burden.
I propose that we develop a concrete plan for the removal of NCL. Something along the lines of
@ESMValGroup/esmvaltool-coreteam, @ESMValGroup/esmvaltool-developmentteam, thoughts?
katjaweigel
commented
2 years ago There are NCL PRs (e.g. https://github.com/ESMValGroup/ESMValTool/pull/2156, which should be merged because it was described in one of the ESMValTool paper) which have been there kind of forever and I'm afraid nobody has the resources to convert that all to python. There are probably more, where papers are published but the code is not yet in the ESMValTool main? On the other hand it is probably never a good time for everybody to finally drop NCL but gets more complicated to keep it running the longer we wait.
valeriupredoi
commented
2 years ago Hi @zklaus I fully support we migrate from NCL to Python soon, but that is a very resource-intensive task, who gonna do it? Points 1 and 2 are most welcome, afraid 3 is needed, but much harder than it sounds
The conda-forge build is broken again
where do you see that? GA and feedstock tests still passing...
zklaus
commented
2 years ago Hi @zklaus I fully support we migrate from NCL to Python soon, but that is a very resource-intensive task, who gonna do it?
TBD. But one thing is crystal clear: It will become infeasible to have NCL as a conda-forge dependency. The only question is when.
The conda-forge build is broken again where do you see that?
There are two PRs outstanding to rebuild for new versions of Proj and HDF. Both are failing. If they are not fixed, installing NCL from conda-forge will block updating Proj and HDF, which in turn will prevent the installation of dependencies like netcdf and iris when they depend on those. This is some time in the future, for sure, but as you note, we will need some time.
GA and feedstock tests still passing...
Not sure what you are referring to: Tests in this repo and the core one? Our own feedstock? The NCL feedstock?
valeriupredoi
commented
2 years ago There are two PRs outstanding to rebuild for new versions of Proj and HDF. Both are failing. If they are not fixed, installing NCL from conda-forge will block updating Proj and HDF, which in turn will prevent the installation of dependencies like netcdf and iris when they depend on those. This is some time in the future, for sure, but as you note, we will need some time.
Gotcha! I was suspecting it was one of the big deps
Yeah I was referring to our esmvaltool GA tests (which obv don't contain the pkg build test no more) and the esmvaltool feedstock tests
valeriupredoi
commented
2 years ago cheers for raising this BTW, Klaus! It's getting very tight indeed
bouweandela
commented
2 years ago This might be a good point for discussion in the upcoming workshop https://github.com/ESMValGroup/ESMValTool/discussions/2588. At the moment, slightly less than half of all diagnostic code is NCL, but maybe Python and/or R code is just more compact. It seems unlikely that there will ever be enough time to translate all (or even many) NCL diagnostics to Python. Maybe the @ESMValGroup/scientific-lead-development-team could think about which diagnostics are required as a minimum for the tool to remain interesting from a scientific point of view?
I do agree that it would be good to stop introducing new NCL diagnostics in the tool, as seems a waste of time in the longer run.
If installation from conda really becomes a problem and dropping support for NCL is just not feasible from a scientific point of view, an alternative to completely dropping support for NCL would be to just no longer install NCL from conda. The installation of the NCL interpreter would then be left to the users. They might be able to install the NCL interpreter
apt get install ncl-ncargmodule load ncl it if they're on a cluster
axel-lauer
commented
2 years ago I am not really sure I fully understand why some people are so eager to get rid of NCL. Even if installation from conda gets too difficult to maintain, there are other options as @bouweandela pointed out.
At the moment, 17 recipes with about 100 diagnostic scripts are based on NCL. Among those are some of our flagship diagnostics such as IPCC AR5, SMPI and perfmetrics. I do not see any realistic possibility to convert so many diagnostics to Python. When transitioning from ESMValTool v1.1 to v2.0, a lot of diagnostics could not be converted because it was way beyond our resources (even though we had quite a few coding workshops). And that did not even involve changing the original programming language. Dropping NCL would change the ESMValTool significantly from what has been advertised over the last years and for this reason alone, I am strongly advising against such a step. In the end, it does not matter too much if people would have to install NCL on their own, the main point is that diagnostics such as IPCC AR5 are in and could (at least in theory) be used. That said, I also find it difficult to simply decide to trash the work of quite a few contributers over the last years just like that.
remi-kazeroni
commented
2 years ago Note that flagship diagnostics will soon also include code used for IPCC AR6 (first PR in #2533). I think a majority of the ESMValTool produced figures for AR6 are based on NCL diagnostics (new ones but also already existing ones). I guess here as well, a translation of the diagnostics to Python is not feasible in practice... It would be quite unfortunate to continue advertising ESMValTool for the produced AR6 figures and at the same time not being able to merge the code to the main branch and not being able to use the Tool to reproduce these figures because support for NCL was completely dropped... I support the idea of considering alternative ways to install the NCL interpreter if such installation from conda becomes problematic.
zklaus
commented
2 years ago I think there is some misunderstanding. It certainly is not my intention to "trash" anybody's work. Quite the contrary, I will come back to that. But it is my assessment that NCL will become unavailable. That is not due to some weird behavior by conda-forge, but because NCL's code quality is not up to modern standards and because it depends on other libraries that are still developing and that we need to keep updated, such as Proj. As for alternate installation methods, note that they all will face similar issues as conda-forge. For apt, you can follow the bug trackers for the debian package and the ubuntu package (FTBFS means "fails to build from source"). Currently, DKRZ does not offer an NCL module on Levante, though they certainly may (be willing to) change that as Levante develops.
Even if things can be kept running, it will never be made to parallelize or updated to deal with CF 1.10 or UGRID, which means that we risk breaking another promise of ESMValTool. After all, the AR6 recipes (just as an example) are not interesting to produce the exact same figures from the exact same data---those are published---but rather because it is easy to add CMIP7 and CMIP8 models in there. Will NCL diagnostics be able to handle that? I doubt it.
Will all of this happen overnight? No. Within one year? Probably not. Two? Perhaps. Three? Probably.
Some of this may be mitigated by maintenance. However, NCAR's GeoCAT team does seem to take a rather hands-off approach to maintenance. Have a look at the NCL issue tracker and think whether you want to rely on that level of support. Also note, that the two packages touted at the time as a replacement and bridge to Python, PyNGL and PyNIO also have since been placed into "maintenance mode" and require patches just to make them compile. It seems currently the CI on the official NCL repo is not able to build NCL. Despite conda being the officially suggested mode of installation and the promise to make bugfix releases and keep the conda install working, you have to go back to May 2020 to find contributions from the NCL team to the conda build.
The community has tried to apply enough duct tape to hold things together. I myself have spent weeks to keep NCL and PyNGL running in the current conda-forge. We can keep this up for a bit longer, but I won't be doing that in 5 years.
What I am saying is: With NCL, we are on a train that is slowly falling apart. We still have time, let's salvage what we can by making a plan now. Let's not wait until it just disintegrates below us.
So returning to what I said at the top: I don't want to trash the work. I don't want to lose those diagnostics. That is precisely why we should think about how best to deal with them now. Then we will have some time and we will be able to preserve the most important diagnostics and recipes for use with CMIP7, 8, and beyond. Perhaps they will even acquire a few new tricks along the way.
zklaus
commented
2 years ago In case we need to contact NCL developers, the following is the complete list of users that have touched .ncl code in the ESMValTool repository. I have categorized them as possibly active, known retired (from ESMValTool), and technical (people that are not known as NCL developing scientists, but rather have been in contact due to housekeeping. I have erred on the side of counting people as active when I didn't know for sure, though likely not everyone on the active list is actually still active.
Edit: To clarify, this is git authors of ncl files currently in the main branch. It does not capture branches that are not integrated yet and it does also not capture possible authors/collaborators outside of git, which particularly means authors of ESMValTool version one diagnostics that have later been ported by someone else. This seems reasonable to me because those are old contributions and absent further involvement since then, we probably don't expect those contributors to resume maintenance of their code.
katjaweigel
commented
2 years ago Although I'm not writing new software in NCL, I'm involved in pull requests containing NCL as a maintainer for a tool from @irenecionni (https://github.com/ESMValGroup/ESMValTool/pull/2156) and there is a PR planned for things from the AR6 repository, where I build up on an existing NCL cmorizer. In the second case I maybe could convert it to python, for the PR https://github.com/ESMValGroup/ESMValTool/pull/2156 I don't see that I, or @irenecionni would have the resources to change these diagnostics to python.
Peter9192
commented
2 years ago We still have time, let's salvage what we can by making a plan now.
Could it be an interesting internship/MSc graduation project?
remi-kazeroni
commented
2 years ago To clarify, this is git authors of ncl files currently in the main branch
Could you maybe also give estimates of how many NCL functions and lines of code we have in the main branch? It would give an idea of the effort to translate NCL code into Python given the number of persons listed above. But this would not count NCL code waiting to be merged in some PRs (e.g. #2156), or AR6 diagnostics... So this effort would certainly be larger than that.
zklaus
commented
2 years ago Here is the number of all non-empty non-comment lines of all ncl files in diag_scripts except for cvdp and shared:
| number of lines | diag_script |
|---|---|
| 66 | xco2_analysis/stat.ncl |
| 72 | examples/diagnostic.ncl |
| 83 | bock20jgr/corr_pattern.ncl |
| 83 | ipcc_ar5/ch09_fig09_6.ncl |
| 96 | carbon_ec/carbon_aux.ncl |
| 96 | perfmetrics/cycle_zonal.ncl |
| 98 | perfmetrics/cycle.ncl |
| 100 | perfmetrics/cycle_latlon.ncl |
| 116 | russell18jgr/russell18jgr-fig5.ncl |
| 119 | russell18jgr/russell18jgr-fig2.ncl |
| 132 | russell18jgr/russell18jgr-fig7h.ncl |
| 157 | carbon_cycle/two_variables.ncl |
| 158 | perfmetrics/zonal.ncl |
| 158 | russell18jgr/russell18jgr-polar.ncl |
| 160 | russell18jgr/russell18jgr-fig7i.ncl |
| 165 | mder/select_for_mder.ncl |
| 171 | perfmetrics/latlon.ncl |
| 171 | seaice/seaice_aux.ncl |
| 173 | ipcc_ar5/ch09_fig09_3.ncl |
| 180 | russell18jgr/russell18jgr-fig5g.ncl |
| 199 | carbon_ec/carbon_beta.ncl |
| 202 | seaice/seaice_trends.ncl |
| 205 | ipcc_ar5/ch12_map_diff_each_model_fig12-9.ncl |
| 205 | ipcc_ar5/ch12_plot_ts_line_mean_spread.ncl |
| 208 | austral_jet/asr.ncl |
| 208 | xco2_analysis/carbon_plots.ncl |
| 210 | russell18jgr/russell18jgr-fig3b.ncl |
| 211 | russell18jgr/russell18jgr-fig3b-2.ncl |
| 220 | russell18jgr/russell18jgr-fig9a.ncl |
| 225 | carbon_ec/carbon_constraint.ncl |
| 225 | russell18jgr/russell18jgr-fig9c.ncl |
| 232 | russell18jgr/russell18jgr-fig9b.ncl |
| 233 | bock20jgr/model_bias.ncl |
| 235 | carbon_ec/carbon_gammaHist.ncl |
| 237 | clouds/clouds_interannual.ncl |
| 251 | seaice/seaice_tsline.ncl |
| 256 | xco2_analysis/delta_T.ncl |
| 262 | bock20jgr/tsline_collect.ncl |
| 267 | ipcc_ar5/ch12_calc_map_diff_scaleT_mmm_stipp.ncl |
| 268 | eyring06jgr/eyring06jgr_fig01.ncl |
| 272 | eyring13jgr/eyring13jgr_fig12.ncl |
| 273 | clouds/clouds_bias.ncl |
| 274 | perfmetrics/main.ncl |
| 276 | russell18jgr/russell18jgr-fig4.ncl |
| 280 | eyring06jgr/eyring06jgr_fig05b.ncl |
| 282 | xco2_analysis/sat_masks.ncl |
| 284 | seaice/seaice_ecs.ncl |
| 288 | ipcc_ar5/ch09_fig09_6_collect.ncl |
| 291 | eyring06jgr/eyring06jgr_fig05a.ncl |
| 296 | xco2_analysis/panel_plots.ncl |
| 316 | ipcc_ar5/ch12_calc_IAV_for_stippandhatch.ncl |
| 317 | ipcc_ar5/ch12_plot_zonal_diff_mmm_stipp.ncl |
| 324 | mder/absolute_correlation.ncl |
| 329 | bock20jgr/corr_pattern_collect.ncl |
| 337 | ipcc_ar5/tsline.ncl |
| 346 | ipcc_ar5/ch12_plot_map_diff_mmm_stipp.ncl |
| 354 | carbon_ec/carbon_co2_cycle.ncl |
| 358 | carbon_cycle/mvi.ncl |
| 360 | perfmetrics/collect.ncl |
| 365 | seaice/seaice_yod.ncl |
| 366 | ipcc_ar5/ch12_calc_map_diff_mmm_stippandhatch.ncl |
| 376 | ipcc_ar5/ch12_calc_zonal_cont_diff_mmm_stippandhatch.ncl |
| 379 | carbon_ec/carbon_tsline.ncl |
| 388 | xco2_analysis/station_comparison.ncl |
| 398 | bock20jgr/tsline.ncl |
| 398 | eyring06jgr/eyring06jgr_fig15.ncl |
| 411 | carbon_cycle/main.ncl |
| 433 | emergent_constraints/snowalbedo.ncl |
| 433 | ipcc_ar5/ch12_ts_line_mean_spread.ncl |
| 471 | clouds/clouds_ipcc.ncl |
| 472 | xco2_analysis/global_maps.ncl |
| 480 | ipcc_ar5/ch12_snw_area_change_fig12-32.ncl |
| 499 | xco2_analysis/main.ncl |
| 532 | clouds/clouds_taylor.ncl |
| 535 | russell18jgr/russell18jgr-fig6b.ncl |
| 550 | russell18jgr/russell18jgr-fig6a.ncl |
| 563 | austral_jet/main.ncl |
| 852 | mder/regression_stepwise.ncl |
| 901 | clouds/clouds.ncl |
| 1243 | emergent_constraints/ecs_scatter.ncl |
| 24015 | grand total |
phillips-ad
commented
2 years ago A brief introduction: I am the author of the CVDP, and for many years served as a scientific advisor to the NCL Development Team. Since the dissolution of the NCL Dev Team and the formation of the GeoCAT Dev Team, I no longer am directly involved with NCL/GeoCAT development efforts, and do not speak for GeoCAT. However, I have been following this discussion for a while, and thought it was important to speak up, but only after I discussed the situation with the GeoCAT Team lead @erogluorhan.
Since the dissolution of the NCL Dev Team, I have been assured many times by NCAR/GeoCAT leadership that NCL builds will be supported as long as the community asks for it. There is no doubt at the present time that the community (beyond ESMValTool) still wants supported builds. @erogluorhan has stated providing stable builds via conda is the number one NCL-related priority for the GeoCAT team. Another priority of the team is to support community contributions to NCL. There is no doubt that these priorities have not been treated as such over the past couple of years, and there are 2 principal reasons why: 1 - GeoCAT has not had the personnel to be able to support all of their external packages and to pursue their funded mandate to build new python tools. 2 - Due to retirements and people switching jobs, there is very little NCL institutional knowledge left within GeoCAT.
I find it completely understandable that NCL users and system/package maintainers have expressed frustration with the current situation. Regardless, for now, GeoCAT has agreed to evaluate and fix NCL's conda builds and to address @zklaus proj pull request https://github.com/NCAR/ncl/pull/173, both in short order.
Some other items from the discussion above that I would like to chime in on:
I hope the above provides some clarity with respect to NCL and advances the discussion!
zklaus
commented
2 years ago Thanks, @phillips-ad for your comments. It's nice to hear from the NCL side!
It would be great if the NCAR could pick up the maintenance of NCL again. In that case, we may push the other bugfixes that reside only in the conda-forge feedstock at the moment upstream as well. This may even have to happen before the integration of NCAR/ncl#173 to make it possible again for NCL's CI to build NCL. Please don't hesitate to ask for assistance with that if you would like that. Sustaining this kind of maintenance could really help us to keep things running in the transition phase.
On the other items you raised:
We do make frequent use of CVDP via ESMValTool at my institute, so I am very happy to hear that you are still interested in maintaining it and possibly moving it to Python. Perhaps you might want to consider doing that directly in ESMValTool. This way, you can benefit from a lot of the standard preprocessors for data fixing, regridding, spatial and temporal statistics, and more. Where there are gaps in the preprocessors, adding those needed for CVDP might prove useful beyond that application for recipes and we can likely give a little bit of support in these developments. If you decide that it should stay in a separate package, I personally would still be interested to have a look at a possible Python version.
Thanks again for you input, and do get/stay in touch about anything we can do to help with NCL or CVDP.
erogluorhan
commented
2 years ago Hello all, I am Orhan from the GeoCAT team. @phillips-ad already shared significant insights about our team and NCL commitments, and I'd like to make a few more:
Keeping NCL builds stable via conda is the number one NCL-related priority for the GeoCAT team, but I am sad to say, this is not the priority of the GeoCAT team in its broader roadmap. This is only because we already have several projects that we are funded to develop and maintain in the Python ecosystem for the worldwide geoscience community in an open development approach, and we are both short-staffed and has limited-knowledge on NCL internals. That said, for most of the time we even can't respond to issues and NCL emailing lists (We could actually respond to them saying we received them, but I am not sure if it would be helpful at all since we could still not be able to provide actual timely support after then). This does not mean we won't maintain NCL, but I am aware that we have not been and will unlikely be able to provide timely support on our commitments. That said, the NCL user community is always welcomed to be part of the NCL developers/collaborators list and get things moving forward (either on Github or on email lists).
Regardless of the above clarification, maintaining NCL is still in our agenda for a foreseeable future whenever time permits. Therefore, I am personally working on the repository right now to address critical issues mentioned here and in some other channels to get its users up and using the tool without any issues.
Regarding PyNGL and NCL's graphics power in general, we have developed the GeoCAT-examples plotting gallery that has been quite successful to replicate NCL plots in several categories. Th relation of this development to PyNGL going into maintenance mode was detailed in our November 2020 community update. We have been receiving strong feedback about how GeoCAT-examples is capable of regenerating NCL plots in Python and how it is going above and beyond being replication of NCL, especially nowadays, with some novel plotting examples, e.g. see interactive MPAS plotting example.
zklaus
commented
2 years ago Hi @erogluorhan, it's really good to hear from you, too!
I understand that there are no long term plans for NCL and that it is difficult to maintain such software with limited resources.
I am happy to help feeding the existing bug fixes upstream to keep things running a bit longer. I guess the first question is how to approach the issue of CI. Right now, it seems the CI scripts cannot build NCL, which renders all testing impossible. Clearly, you would like to have the testing before allowing PRs in.
It is possible that some of the *.patchs in the conda-forge feedstock can contribute to fixing this, but it would be good to understand how you want to tackle this, i.e. fix the circleci or pick up your work on github actions in NCAR/ncl#170?
With the CI in place, it should be straightforward to integrate the proj6 port and the boz fixes. From there, we might be able to tackle some of the open issues.
Here is probably not the place to discuss how to fix the NCL CI, but let me know if you want to take it up in the NCL repo or any other way.
erogluorhan
commented
2 years ago Hi @zklaus , yes, let us not clutter this repo much, but just to quickly answer: I am currently working on getting the CI up and running again. My first priority would be to get Github Actions configured as you pointed out my PR (This would also help GeoCAT team address future maintenance needs hopefully more timely in NCL as we have much experience with Github actions). However, if I feel like setting up Github Actions would take much time, I'd also consider fixing th existing CircleCI config just to expedite the things. Let us follow up the rest of this conversation under either NCAR/ncl PR#170 or your NCAR/ncl PR#173.
The discussion about the future of NCL support in the ESMValTool has come up quite frequently in the last weeks in various GitHub issues here.
The reason behind this discussion is (should be) well known: NCAR has decided to stop the development of NCL, which has been put into maintenance mode. Graphics routine have been translated to Python and are available via the PyNGL library. This means that NCL is still available and will be available "for the foreseeable future" (as NCAR itself states in its letter to users), but the transition to Python is recommended.
Some general considerations to trigger the discussion:
@ESMValGroup/esmvaltool-coreteam @zklaus @hb326 @LisaBock @ruthlorenz (please tag other non-core members who might be interested in this topic).