search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
138 stars 51 forks source link

Survey: Frontend Language Support #625

Open ax3l opened 4 years ago

ax3l commented 4 years ago

Dear @openPMD/general-contributors,

this issue is pinned to keep track of your needs for our reference API in terms of minimum supported language features.

So far we support Python 3.6+ and C++14 or newer. But as compilers and languages progress, we would love to drop old language revisions in newer releases to keep maintenance at a minimum and make use of newer language features in public interfaces. Please vote in the following comments according to the minimum requirements you have in your downstream code-bases. This will help us to plan potentially breaking changes with minimum hazzle.

Voting in the C++ and Python frontend will help us to ping and interact with you in case we would like to drop, e.g. C++14 bindings (for C++17 and newer) or older Python bindings in the future.

Voting on new language bindings does not implicate someone will find the time to provide such implementations, but it potentially motivates volunteers to step forward and add further bindings or share their own implementations where needed.

ax3l commented 4 years ago

C++ Frontend

As of today, I (only) need support for

You can update your vote at any time, e.g., if your code-base updated.

Note: C++ compiler defaults and C++ compiler support.

ax3l commented 4 years ago

Python Frontend

As of today, I (only) need support for

You can update your vote at any time, e.g., if your code-base updated.

ax3l commented 4 years ago

C Frontend

We do not yet provide a C-frontend. If one would spend time developing one, I (only) need support for

You can update your vote at any time, e.g., if your code-base updated.

Note: C compiler defaults.

Potential library usage: shroud

ax3l commented 4 years ago

Fortran Frontend

Currently, there are at least two implementations of a HDF5-Fortran openPMD library [ref1] which we hightly recommend to check out and use. We have no plans yet to do so, but just to mention the technical possibilities: one could in principle, based on implementing a C-Frontend first, use the former to implement Fortran bindings herin as well. This would widen the supported backends to include, e.g., ADIOS1 and ADIOS2 for all Fortran users. Alternatively, one could also add these backends via their native Fortran bindings to the aforementioned libraries.

If one would add Fortran bindings to openPMD-api, I (only) need support for

Note: Fortran bindings prior to the 2008 standard are extremely cumbersome (read as: not worth spending time developing), since iso-C-bindings were first standardized in version 2008. Potential library usage: shroud

You can update your vote at any time, e.g., if your code-base updated.

ax3l commented 4 years ago

More Exotic Bindings

It's in principle possible to add bindings for further (modern) languages in case there is a sufficiently large user-base for it. Just tell us what you currently use in production workflows, maybe someone wants to team up with you and build bindings!

I use the following languages in my workflows and I would benefit from adding bindings to

(Multiple answers are welcome, too.)

mccoys commented 4 years ago

I know I should use the front-end already ... Will do when I have time. I envision only c++11 & py3.5+ for now

ax3l commented 4 years ago

Sounds great, feel free to ping us if you need help with anything when adopting to the API (reviews, more examples, etc.). Since Python 3.5's EOL is in Q3 2020 I do not expect any drop of our supported versions soon until it runs into this. Conda-Forge already transitioned to ship Python 3.6+ only, but that's just one of our many release channels.

ax3l commented 4 years ago

Announcement: since most user-level distributions already dropped Python 3.5 and it will not receive any updates after September 2019, we anticipate to drop Python 3.5 support in upcoming releases.

This approach is on par with other major projects such as Numpy dropping Python 3.5 in upcoming releases (1.19.0+). We continue to support Python 3.6-3.8 and anticipate addition of Python 3.9 after its October release.

Please see: https://devguide.python.org/#status-of-python-branches

Update: Python 3.5 dropped and 3.9 added in #828 (0.13.0 release). We already ship Python 3.9 via conda-forge for 0.12.0.

williamfgc commented 4 years ago

I see a label for Julia, are you guys invested on it? I'm currently evaluating its use and seems very promising now that it's been stable for a year.

ax3l commented 4 years ago

Unfortunately not, I know exactly one person in our community that does Julia @berceanu

bussmann commented 4 years ago

Hi William, yes, Julia support would be lovely. As ax3l pointed out, community-wise the interest is unfortunately limited, so the manpower is as well. Julia is IMHO quite o.k., but the parallelism does not come out of the box as promised (it never did). Why would you got for Julia? What would be the features you are missing with Python?

williamfgc commented 4 years ago

@ax3l @bussmann thanks for chiming in. Python is great, nothing is missing with the added ecosystem (numpy, scipy, matplotlib, etc.), still dependency versions must be tracked when packaging with conda, pip, etc.. We are mostly exploring Julia for the long run (still young in 2020, I just hit a wall myself with IDE support) as it presents interesting concepts built into the language (type declarations, the parallelism you mentioned, math) targeting the kind of work we do in their roadmap. I just got mostly curious as I saw the label in this issue and I wanted to learn more about your view/experiences. Thanks.

ax3l commented 3 years ago

With our main downstream consumers now upgraded to C++14 code bases, we'll soon migrate to C++14, too. This will simplify the maintenance of the library and we can stop testing some ancient compilers that only came with now deprecated operating systems.

Many compilers already switched to build with C++14 by default and some already set C++17 as their default compilation language. C++ is generally forward-compatible and we make sure in our CI that our C++ standard (e.g. 14) requirement will built with C++17 and 20 flags, too. We will generally aim to continue to adopt new C++ standards, as soon as vendor compilers and HPC systems provide stable support for them; and thus once our main downstream users can make the switch.

Update: transition to C++14 and newer will happen with the 0.13.0 release #825

eschnett commented 3 years ago

@axl3 Incidentally, I started working on a Julia package openPMD_api_jll today.

eschnett commented 3 years ago

C++: C++17 Python, C, Fortran: n/a Julia: yes please!

ax3l commented 3 years ago

Awesome, thanks for the update & contributions @eschnett! :)

ax3l commented 2 years ago

C++ Frontend Announcement: With WarpX and PIConGPU both upgrading to C++17, we will likely transition to require C++17 #1164 in openPMD-api with release 0.15 and newer :tada:

ax3l commented 2 years ago

Python Frontend Announcement: Since Python 3.6 is EOL in three days, the last release line that will support it is 0.14.*. It's likely that major user-level package managers, like conda-forge, and release support software, like cibuildwheel, will drop support soon after (to make room in build resources for newer releases, like Python 3.10). https://endoflife.date/python

ax3l commented 2 years ago

@skuschel do you still use Python 3.6 in production? Just checking in, feel free to update your current needs here by updating your reaction emoji :)

berceanu commented 11 months ago

Hi William, yes, Julia support would be lovely. As ax3l pointed out, community-wise the interest is unfortunately limited, so the manpower is as well. Julia is IMHO quite o.k., but the parallelism does not come out of the box as promised (it never did). Why would you got for Julia? What would be the features you are missing with Python?

A short update regarding Julia. While the maturity and diversity of the ecosystem still lags somewhat behind that of Python, I feel that the community has progressed a lot in recent years, especially on issues related to parallelism and GPU kernel generation. Here are two recent comparative papers:

Probably @williamfgc has more details on this, as first author on both studies

DavidSagan commented 11 months ago

@williamfgc Note: There is a Bmad (https://www.classe.cornell.edu/bmad/) port to Julia project in development and part of this project will be a Julia front end for the EXT_BeamPhysics.md extension.