overlap with the climlab package

[brian-rose]

Hi @msmithsm, I wish I had found this earlier because I put a lot of parallel effort into developing Python wrappers for RRTMG_SW and RRTMG_LW for my climlab project. I encourage you to take a look!

It might be a good idea to team up on a robust and portable Python implementation of RRTMG that could be imported into climlab and other projects.

[msmithsm]

@brian-rose, great to hear from you! I'll check the climlab package. I have to admit I'm surprised you found this RRTMG wrapper. I would love to team up and make a more portable implementation. This solution worked for several of my devices, but it requires a fair bit of install time to get the compiler flags right and make sure the dynamic libraries are installed properly. I'd love to hear your ideas and we can get started.

Also, I don't know if you are planning on attending PyCon in May, but I will be there. It would be great to meet and chat in person.

[brian-rose]

Well, finding it is just a matter of searching for RRTMG on github ;-)

I've been tinkering with a few different tools. f90wrap transforms Fortran 90 code into a form that is more robustly digested by f2py. I'm also tinkering with conda for automating the build process. But I haven't gotten very far.

I would be keen to hear if you are able to build climlab (which includes RRTMG) out of the box on mac and/or linux.

[brian-rose]

I was able to get climlab (including RRTMG_SW and RRTMG_LW) to build for Mac and Linux using conda-forge. So there is now one-click installation for a Python wrapper for RRTMG, no compilers required :-)

[brian-rose]

Edit: climlab is available for Windows too!

[brian-rose]

Ping @msmithsm , just checking if you're still interested in maintaining this?

I think the solution I came up with within climlab (building with conda and conda-forge) works quite well and makes for a very portable RRTMG implementation. But it's become clear that it would make more sense to cleave off the RRTMG wrappers from the rest of the climlab code base and maintain them as a separate package. Since I already have working conda build recipes, this would be very easy to do.

Just don't want to step on anyone's toes, and frankly, pyRRTMG is the most sensible name for such a package :-)

[msmithsm]

@brian-rose

That's a really good question. It appears that the f2py package used to build this RRTMG wrapper is not currently being developed, and is a lot slower and less convenient than Cython. It would be a good idea to try to modernize this package, since I haven't touched it for over a year.

[brian-rose]

In my experience f2py works well but suffers from poor and uneven documentation. I don't see that situation changing any time soon.

If you're thinking of pushing this forward, I suggest taking a look at the climt package, which uses Cython to build a RRTMG wrapper.

Since RRTMG is so widely used, I think it would be a real asset to the community to have something like pyRRTMG as a robust stand-alone package with a lightweight wrapper that's easy to build and deploy. This could then be a dependency of larger projects like climt and climlab, leading to less duplication of effort and more reproducibility of modeling results :-)

ping @JoyMonteiro, @mcgibbon, any thoughts on this?

[mcgibbon]

Right now (not being familiar with pyrrtmg) I would suggest to the average user who wants to use RRTMG to install climt and use its RRTMG wrapper. So yes, it's a great starting point.

It sounds like what you may want to do is copy-paste the code out of CliMT and write your own wrappers on the CliMT objects to expose your own interface. I would strongly encourage wrapping the existing interfaces instead of refactoring the code, so that if improvements are made to the CliMT wrapper, you can easily copy them over to your version, and vice-versa. Or so that CliMT can depend on and import from the RRTMG package.

I think for CliMT to depend on an external package instead of containing its own RRTMG wrapper, that package should meet the same usability standards that CliMT tries to meet (regarding documentation, conda installation, platform portability, style, integration testing, etc.). I think those standards make CliMT pretty suitable as a dependency for other projects. You're perfectly free to take the CliMT wrapper as a starting point, and we can look down the line at whether to make that dependency. There are good reasons to make a stand-alone RRTMG package.

[JoyMonteiro]

Yes, you are most welcome to use the wrapper in climt and build a package. I agree with @mcgibbon about criteria for climt to depend on any package. I must note that wrappers within climt include code to set constants and other configuration information from Python, and they are not "hands-off" wrappers.

Looking ahead, it might make more sense to build a package which includes other state-of-the-art radiative transfer codes (Socrates, for instance) instead of one package per library.

[mcgibbon]

Yes, I would love to see a package that has Sympl components for multiple radiative transfer solutions, possibly alongside other interface options.

[brian-rose]

A package that bundles multiple radiation codes would be great, but rather different from what I was suggesting above.

My proposal is that we create a very lightweight Python wrapper for RRTMG that follows the Fortran interface as closely as possible. It basically just solves the problem of packaging and building the Fortran code so that it is callable from Python, and no more than that.

This would be appealing to a broader community of RRTMG users who already have some familiarity with the code but want an interactive implementation of it -- or want to use it as a starting point for building other kinds of interfaces to it, such as CliMT or climlab.

As for @mcgibbon's list of criteria for external dependencies, this is totally sensible, but also (in my opinion) relatively easy to achieve in this case. Both because it is a very lightweight wrapper, but also because there is already in this thread a significant pool of dev talent who can help maintain it to high quality standards!

[JoyMonteiro]

Yes, I understand. If all this entails is a cython wrapper over RRTMG, then anyone interested in maintaining such a package is more than welcome to use what is available in climt. I would have done it myself, but other commitments won't allow this until the summer.

[mcgibbon]

@JoyMonteiro, you've already written what @brian-rose is asking for. You can find it here for longwave and here for shortwave. The _rrtmg_lw.so and _rrtmg_sw.so files generated by CliMT are a very lightweight Python wrapper for RRTMG that follows the Fortran interface as closely as possible.

The main issue with them is that they have quite an unwieldy interface, but that is because RRTMG has an unwieldy interface. IMO, CliMT offers a lightweight wrapper on RRTMG that is much clearer and easier to use than one which "closely follows" the Fortran interface. You can use components in CliMT outside of an interactive model context.

[msmithsm]

Correct. pyrrtmg also follows the fortran interface. I didn't include the .so files because I run cross platform and they are seriously un-portable. The build system I use is poorly documented, but I tested it on Mac and Linux. The build system uses f2py to build the shared libraries that expose the fortran methods as they are in python. These are then linked in the PYTHONPATH. It also includes a function test to verify it was installed correctly. Aside from tests, pyrrtmg has very low overhead. I designed it to be a completely separate package from the model that I wrote it for.

The real issue, as @mcgibbon said, is that RRTMG is not very easy to use outside a model context.

[brian-rose]

Actually I have also written what I'm asking for -- namely a lightweight wrapper for RRTMG than follows the Fortran interface closely. In my case, it's buried inside climlab and built using f2py on conda-forge, which has allowed me to get it working on Mac, linux and even Windows.

But this is essentially my point: there is wasted effort and wasted resources. Three projects have accomplished the same thing in slightly different ways, and now require the entire RRTMG to be rebuilt every time we send our respective codebases through a CI service. For me this is a serious concern, because building RRTMG is slow, and it's getting in the way of the other development I want to do on climlab. I thought (still think) it would be advantageous to us and the community as a whole to converge on a single simple wrapper that we can collectively maintain as needed -- but mostly just get out of the way so we can focus on more interesting stuff.

I'm all for people designing and documenting better interfaces to RRTMG that suit their uses, but serious scientific users and model developers are still going to have to look at the Fortran source from time to time. Short of rewriting RRTMG from scratch, I would argue that keeping the basic wrapper as consistent with the existing Fortran interface as possible makes this work easier. And that seems to be what we've all done anyway. Converging on a common solution for the lowest-level wrapper doesn't stop any of us from coming up with (potentially) better user interfaces to sit on top of it.

It's not necessary that we agree on any of this. I might just go ahead and do this myself -- cleave off my RRTMG wrapper from climlab into its own package. Or perhaps use @JoyMonteiro 's Cython solution instead. (would be interesting to benchmark the two and see if there are performance differences). Not a very sexy package but a useful one.

[mcgibbon]

I mean, we're in agreement @brian-rose. I think we're over-thinking this. If someone (you? @JoyMonteiro ?) makes a good simple wrapper package, the community will collectively converge on it if they think it's better than the alternative of maintaining their own wrapper. We don't need a huge coordinated effort to make this happen (we've already done it separately), and the maintenance work going forward should be minimal. I prefer having Cython wrappers out of fears of F2PY not being supported in the future and generally being confusing to use in my experience, but you can use whichever solution. It should be easy to extract from CliMT (like I said, you just need the part that makes _rrtmg_lw.so and _rrtmg_sw.so).

Like you said, the CI time saved alone makes it worthwhile for you to do a split like this for your package, and the ecosystem can evolve from there.