Making specutils compatible with data sets from fields besides astronomy

[namurphy]

I'd like to raise a rather broad issue for discussion about making specutils compatible with spectroscopic data sets from fields such as laboratory plasma physics and heliophysics.

I'm coming from PlasmaPy, which is a young package that is striving to do for plasma physics what Astropy is doing for astronomy. Laboratory plasma experiments often have spectroscopic diagnostics. It would be wonderful to be able to use the same code to analyze remote observations, output from laboratory plasma diagnostics, and synthetic spectra from numerical simulations. I'm also hoping to reduce unnecessary duplication of functionality since we have begun a separate package called SpectroscoPyx. (edit: SpectroscoPyx is no longer under development) We've also had related discussions within the Python in Heliophysics Community too.

There are a couple of challenges that I can think of. It's pretty uncommon for laboratory plasma experiments to make experimental data openly available. I also don't know of any open metadata schemas for spectra.

I don't do much spectroscopy myself so I don't have use cases yet, though I've been intending to collect some from the plasma community. Probably the most common data formats are NetCDF and HDF5 (though the tokamak community uses MDSplus). I'm wondering if creating a standard way of representing general spectra in HDF5 files and making them able to be read in using specutils would be enough.

This issue came about from an unconference session in Python in Astronomy 2019, at the suggestion of @nmearl. I'm curious about other people's thoughts on this. My sincere thanks to everyone who has contributed to this project!

[dhomeier]

I am probably something of a semi-outsider myself, working in atmosphere modelling and the creating of synthetic stellar spectra. Meaning I am less interested in typical functionalities like spectrum extraction, reduction, EW measurements etc., but for something that helps with storage of the computed spectra in a standard format and facilitates standard operations like rebinning, degrading resolution, wavelength shift, or providing input to synthetic photometry. So far I've been hesitating to refactor my own bunch of (rather poorly organised) tools based on one of the existing spectroscopy packages, as there have been a few coming and going, probably starved for lack of manpower. But with APE13 Specutils and Specviz are hopefully now on their way to become the standard spectroscopy tools in the Astropy ecosystem. Supporting different data formats (my workflow is also based on HDF5 model containers) by adding their own default_loaders should not be too difficult as long as there are already low-level Python libraries. HDF5 for example can already be read in quite transparently by Astropy's Table interface. However on the astronomy side the most supported new format is now ASDF, so I'd expect development to concentrate on that.

[nmearl]

I'm wondering if creating a standard way of representing general spectra in HDF5 files and making them able to be read in using specutils would be enough.

Is this the main challenge? Creating loaders for data formats expected from the field, or are there associated extensions to the Spectrum1D object that'd need to be made in order to accommodate the data format?

[hcferguson]

Reading the HDF5 file itself is easy with the standard library, but I don't think there are any metadata standards to tell you how to interpret spectral data. For now I would suggest pointing people to the ability to read tabular HDF5 data into Astropy tables, where they can then inspect and feed the right columns when creating a Spectrum1D object.

NetCDF is a lot more like FITS and ASDF in having metadata standards (conventions), as well as support for world coordinate systems. I don't think it is used in Astronomy, although I found some links to solar irradiance data when I searched.

[pllim]

@namurphy , did the replies here or the custom readers/writers machinery help? Or did you end up using something else?

[namurphy]

Alas, we haven't made much progress in functionality for spectroscopy of laboratory plasma experiments over at PlasmaPy. The main barriers are that we haven't had contributors who have been able to focus on it, and the dire lack of metadata standardization. With respect to the latter, I've been pushing the plasma community to adopt metadata standards, so hopefully the situation will improve in the next few years. If we do end up having the capacity to work on it, we'll be sure to look into specutils more.

[pllim]

Thanks for the update. Good luck! 😸