search

NOAA-EMC / NCEPLIBS-bufr

The NCEPLIBS-bufr library contains routines and utilites for working with the WMO BUFR format.
Other
46 stars 22 forks source link

Feedback on conda-forge recipe for NCEPLIBS-bufr #589

Closed darothen closed 6 months ago

darothen commented 6 months ago

Hi all,

I'm working on a data engineering project involving processing data from BUFR files at very large scale with a package built using the NCEPLIBS-bufr Python interface. Currently I'm considering the deployment options for my project and publishing a recipe to conda-forge for building the NCEPLIBS-bufr seems to be the simplest option. Note that at least three other NCEP libraries have previously been packaged for dissemination via conda-forge: NCEPLIBS-ip, NCEPLIBS-sp, and NCEPLIBS-g2c. Note that these packages get a non-trivial amount of use, with the first two packages having about 2,500 downloads each and the g2c library over 16,000 downloads.

I've taken the liberty of creating a recipe at conda-forge/staged-recipes#26148. I can volunteer as the maintainer of this recipe. However, before I contribute the recipe I want to make sure that there aren't any NOAA policies or issues that I should be aware of that might govern the re-distribution of the code, and to make sure that the development team is okay with me proceeding with this effort. I don't expect any resources or time would be needed from the team except possibly for a few random questions.

Thanks in advance. Happy to discuss any issues here, if there is a path forward.

jbathegit commented 6 months ago

@AlexanderRichert-NOAA @edwardhartnett @aerorahul

Any thoughts on this?

AlexanderRichert-NOAA commented 6 months ago

I don't have too much to add here. As @jbathegit mentioned, the code is all available as-is without warranty, so I can't think of any limitations on usage or redistribution that would be relevant here.

For whatever it's worth, NOAA EMC and EPIC use Spack for deploying libraries and utilities, so depending on your use case, a lot of the work is already done for you in terms of writing/maintaining package recipes (the main trade off being that conda is almost certainly more popular than Spack, so the pro's/con's of each might depend in part on your target audience/platforms).

darothen commented 6 months ago

Thanks, @AlexanderRichert-NOAA!

I'm familiar with Spack but I haven't seen it penetrate much in the communities I typically work with. On the other hand we have pretty robust tooling for working with conda, and since NCEPLIBS-bufr works just fine with gfortran, I think it's probably the way to go for the initial thrust of this project.

Thanks for your help! There's some linked discussion at conda-forge/staged-recipes/pull/26148 where I'll make some decisions about handling failing test cases that you may want to be aware of but shouldn't impact anything moving forward. TL;DR there's a quirk with how some of the provided tests build when the entire library is packaged as a shared library on OSX, but since the core library test coverage is fine I'll likely just comment out those tests rather than deal with pushing any fixes to the tests themselves upstream.

jbathegit commented 6 months ago

Closing this issue in the NCEPLIBS-bufr repository. Any further collaboration will be documented within the conda-forge repository.

darothen commented 6 months ago

FYI this is now delivered at conda-forge/nceplibs-bufr-feedstock - you should be able to install on x86_64 or osx64/osx-arm64 using the following recipe:

conda create -n bufr_test_env -c conda-forge python=3.11 nceplibs-bufr
python -c "import ncepbufr; print(ncepbufr.__version__)"

I will update when v12.1.0 of this library is released. I can probably add Python 3.12 support at that point, too, since the latest version of library build system no longer uses distutils.