Relation to specutils

[olebole]

Hi,

I am currently reviewing Pypelt for the inclusion as an affiliated package. For me, the relation of the package to the specutils package (an Astropy coordinated package) is unclear. There seem to be some duplicated efforts, f.e. the OneSpec class seems to have a similar goal as the Spectrum1D class of specutils.

Is there a specific reason not to use specutils? I could even imagine that some routines (like spectrum coadding) may be better put into specutils.

[profxj]

In short, PypeIt was developed in parallel with the new Spectrum1D (there was an original but it was deemed undesirable).

Second, the functionality we desire for OneSpec is much lighter than what is built into Spectrum1D (e.g. uncertainty handling).

All that said, I could imagine a day where the two paths merged, but that's a bit hard to predict.

As a short-term solution, I presume it would be easy enough to add a to_spectrum1d function in OneSpec if users requested it.

[olebole]

Would it be useful to contact the specutils people to see how a common class hierarchy could look like? To me, having two independent classes in the Astropy ecosystem for basically the same thing seems not the best way forward.

[profxj]

Well, there is at least even a 3rd in the ecosystem:

https://github.com/linetools/linetools/blob/master/linetools/spectra/xspectrum1d.py

which existed well before the newer development of Spectrum1D. I think specutils had special needs in mind (JWST) and went off on their own path.

[olebole]

Maybe someone from specutils, @rosteen or @nmearl could comment here?

[rosteen]

Any decisions on this front were made before my time, but as far as I'm aware the needs of pypeit were at least on the radar when the current specutils was designed, so future merger/cross utility seems perfectly reasonable. We certainly didn't design specifically around JWST, there are data loaders for JWST included in specutils (largely contributed by people on the JWST team) but there are also loaders for a bunch of other missions/datasets as well.

@eteq probably has a lot more historical context on the decision making than I do.

[nmearl]

specutils, at its core, is meant to be an analysis package. That is, spectral data should be reduced and properly formatted before being fed into specutils. From there, the package modules represent ways to interact with and analyze the spectral data.

Comparatively, PypeIt is more a reduction package: it can take raw observations and detector-specific information and reduce it to the level that would be useful in a scientific analysis. It's understandable that PypeIt would need something to represent their 1D spectral data, but not necessarily something that would need to interoperate deeply with astropy-oriented analysis functionality (e.g. astropy model fitting, complex wcs operations, uncertainty propagation, etc).

I feel that PypeIt and specutils do operate in distinct spaces, but with some overlap that I think could be easily addressed in the future as @profxj points out.

[astronomeara]

Here at Keck, I’m seriously looking at incorporating many of the specutils/specviz things for quick-time analysis. What would be nice is if we could eventually guarantee that PypeIt generated data can be easily read in, or at least we can make a translator.

J

On Sep 16, 2021, at 7:55 AM, Nicholas Earl @.***> wrote:

specutils, at its core, is meant to be an analysis package. That is, spectral data should be reduced and properly formatted before being fed into specutils. From there, the package modules represent ways to interact with and analyze the spectral data.

Comparatively, PypeIt is more a reduction package: it can take raw observations and detector-specific information and reduce it to the level that would be useful in a scientific analysis. It's understandable that PypeIt would need something to represent their 1D spectral data, but not necessarily something that would need to interoperate deeply with astropy-oriented analysis functionality (e.g. astropy model fitting, complex wcs operations, uncertainty propagation, etc).

I feel that PypeIt and specutils do operate in distinct spaces, but with some overlap that I think could be easily addressed in the future as @profxj https://github.com/profxj points out.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pypeit/PypeIt/issues/1263#issuecomment-921111546, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNJKTD7CCOVLB2T7S4DIL3UCIVRXANCNFSM5DZQFT7Q.

[eteq]

I feel that PypeIt and specutils do operate in distinct spaces, but with some overlap that I think could be easily addressed in the future as @profxj points out.

I mostly agree with that, but I think it is worth thinking about how/if the machinery in pypeit can use the specutils data structures internally (echoing what @rosteen said that this). That doesn't necessarily have to be the case for affiliated package inclusion, but it's something to think about (and historically in some cases has landed affiliated packages in "yellow" instead of "green" status, although I think @olebole can make his own decision there!)

Where I think there is more overlap is pypeit and the specreduce package. That said, I think those also have somewhat different aims - pypeit aims to be a all-in-one reduction process for specific (albiet many) spectrographs, whereas specreduce is almost exactly the opposite in philosophy - intended to be a toolkit one can pull together pieces of to make a pipeline, but not be the pipepline itself. If we think of it that way, there's room for the two to grow together over time, as long as lines of communication stay open. And in the meantime, pypeit is certainly more immediately useful for spectrographs it supports than specreduce is at least as of right now!

[profxj]

I propose the moment we hear more people are using specutils to view/analyze their PypeIt products than linetools then we abandon the latter. I surely have no desire to maintain its GUIs indefinitely. ;)

[olebole]

@eteq specreduce isn't an Astropy affiliated package, is it?

To me, the situation currently looks as if there are two competing approaches to handle spectra: one is represented by specutils, specreduce and synphot (driven by STScI), and the other is linetools+PypeIt. And both seem to have their potential and their followers.

This is -- in my opinion -- not a very fortunate situation, as spectroscopy is one of the basics in Astrophysics, and I would expect from Astropy to evolve into a powerful, coherent solution here, where the astronomer (or science sw engineer) does not need to decide for on which of the packages they base their own tools, but to combine them seamlessly. What do you think, @profxj @rosteen @eteq (and all others) is there a chance (and an interest) to unify the packages in the way that they are part of a common (bigger thought) spectroscopy area in Astropy?

[profxj]

I think the ideal answer is "maybe" to unification while the reality is that I have little motivation to develop linetools any further (beyond maintaining what is there).

And I suspect I'd be more likely to adopt speclite in PypeIt if we were to use a different external package. https://github.com/desihub/speclite and DESI's tools for spectra viewing (although I admit I've not used them much so far).

While I agree that spectroscopy is a "basic", I fear the requirements of various users varies to the extent that it is tough if not impossible for one size to fit all..

FWIW, here is an ongoing poll in the PypeIt Users Slack:

[tepickering]

i'll chime in to state that i agree 100% with @eteq that pypeit and specreduce are opposite, yet highly complementary, in their goals and missions.

i also want to state that unity in defining/handling 1D spectra does not currently exist within the astropy affiliated ecosystem. specifically, synphot does not use specutils or Spectrum1D at all and instead defines its own hierarchy of spectrum classes. as with pypeit, the reasons for that are both technical and historical. greater unity and interoperability is obviously desirable, but there needs to flexibility within that for differing needs.