Contribute to Spack NetCDF package?

[rsignell-usgs]

Spack is an awesome tool for installing NetCDF, for example: http://wiki.seas.harvard.edu/geos-chem/index.php/Use_Spack_to_install_netCDF_on_your_system It would be even awesomer if Unidata pushed PRs to Spack when new versions of NetCDF arrive.

Just because I know the NetCDF team is always looking for extra stuff to do. 😄

[edwardhartnett]

Spack is great and I totally agree that it would be fantastic if someone helped keep it up to date. Can you help by showing us how to do it?

[gsjaardema]

I've been doing it for the past several releases. Would not be hurt if someone else wanted to do it.

[rsignell-usgs]

I mentioned it because I know that @WardF is often consulted.

[WardF]

I can take a look at integrating this into our release process; I'll need to take a look. I'm going to tag this as a feature request for 4.8.0, as the rest of my week is taken with administrative tasks that I've been kicking down the road until they can't be, any longer. Once that's done I'll be able to dive in to this. Thanks!

[edwardhartnett]

@gsjaardema what is involved in this? There is interest at NOAA for doing this, and of course there is also PIO which would benefit. Is this hard to do?

[gsjaardema]

If the only change is a new release, I will do something like:

spack checksum netcdf-c

Spack should then ask you how many versions to download and checksum; I usually do a couple just to make sure I get the new and one old (to make sure checksum still matches). It will return something that looks like:

version('4.7.4',   sha256='0e476f00aeed95af8771ff2727b7a15b2de353fb7bb3074a0d340b55c2bd4ea8')

I take that value and add it to the

./var/spack/repos/builtin/packages/netcdf-c/package.py

It is fairly clear where it goes:

class NetcdfC(AutotoolsPackage):
    """NetCDF (network Common Data Form) is a set of software libraries and
    machine-independent data formats that support the creation, access, and
    sharing of array-oriented scientific data. This is the C distribution."""

    homepage = "http://www.unidata.ucar.edu/software/netcdf"
    git      = "https://github.com/Unidata/netcdf-c.git"
    url      = "ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.3.tar.gz"

    def url_for_version(self, version):
        if version >= Version('4.6.2'):
            url = "ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-{0}.tar.gz"
        else:
            url = "ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-{0}.tar.gz"

        return url.format(version.dotted)

    maintainers = ['skosukhin', 'WardF']

    version('master', branch='master')
    version('4.7.4',   sha256='0e476f00aeed95af8771ff2727b7a15b2de353fb7bb3074a0d340b55c2bd4ea8')
    version('4.7.3',   sha256='8e8c9f4ee15531debcf83788594744bd6553b8489c06a43485a15c93b4e0448b')
    version('4.7.2',   sha256='b751cc1f314ac8357df2e0a1bacf35a624df26fe90981d3ad3fa85a5bbd8989a')
...deleted...
    version('4.3.3',   sha256='83223ed74423c685a10f6c3cfa15c2d6bf7dc84b46af1e95b9fa862016aaa27e')

I then just prepare a Pull Request for SPACK and they usually approve and merge fairly quickly. It is best to put the package name netcdf-c: at the beginning of the title of the PR.

This assumes that you have spack installed which isn't too difficult. If the new version adds some new options or dependencies or other changes, then changing the package is more involved.

[gsjaardema]

One good change to make may be to point the download url to the git repository instead of the ftp site (unless that is still the recommended site)

      url      = "https://github.com/Unidata/netcdf-c/archive/v4.7.4.tar.gz"

I think I suggested that several months ago, but the spack developers weren't sure that an 'outsider' should do a change like that which changes the download site and all the checksums...

[WardF]

It is probably an on change to make, but one never knows, let me run it by our web folk to see if there’s a reason not to :)

[edwardhartnett]

OK I know a lot more about spack now, and we are transitioning to spack at NOAA for all our HPC installations. It's a great package installation system and I highly recommend that everyone doing complex software builds, which have many dependencies (perhaps including netCDF), should use spack.

However, distributing netcdf-c on spack is not really a netcdf-c issue.

I don't think that @WardF should necessarily take on any more work to do a release, since maintaining netcdf-c/netcdf-fortran is already more than should be attempted by just two programmers. They are busy enough! Spack enthusiasts like @gsjaardema should continue to maintain the spack releases. (And there are some spack experts at NOAA who will do this if necessary, since netCDF is a requirement for the UFS.) And spack updates happen after a netcdf-c release, so @WardF just needs to concentrate on getting a good release out, and spack will easily be able to cope with it.

@gsjaardema: thank you for the spack work along with your many other contributions. I don't know what kind of mess netcdf-c would be on HPC systems without your help over the decades! You really have been invaluable to the entire netCDF HPC community. Kudos to you!

If there is anything we can do to make the build more simple for spack, let us know. I see that the netcdf-c spack file is quite complex and deals with a lot of quirks of netcdf-c. If eliminating one or more of those quirks is possible, we should do so. @gsjaardema we rely on you to tell us the most effective changes we could make to the build to ease spack deployment.

(However, the spack file will necessarily grow more complex as new features are added to netcdf-c. For example, the addition of zstandard compression will have to be handled in the spack file for the new release.)

Spack seems to have more than reached a critical mass of users. Almost every library or free software package is already well-supported by spack. It's now easier to build complex packages using spack than any other method. Spack is the future.

I believe this issue can be closed.