[Data] Add Time Domain Data to AstroPile

[benboyd97]

Add Time Domain Data to AstroPile

Add time domain photometric data (e.g. PLAsTiCC/Young Supernova Experiment) and spectroscopic data (e.g. Kaepora) to AstroPile.

Contacts: Participants: Ben Boyd, Erin Hayes, Tom Hehir, David Chemaly, Helen Qu

Goals and deliverable

[describe your goals for the week and the deliverables you are aiming for]

Resources needed

[describe the resources (software, skills, data, or just enthusiasm) needed for this project]

Detailed description

Photometry:

• Simulations: PLAsTiCC...
• Real data: SDSS, YSE, ZTF, DES
  • compilations of real SNe Ia (w/ spectra): Pantheon, Pantheon+
  • Kepler (labeled training set @ https://arxiv.org/abs/2107.06301, Full data set?)
  • TESS (Which pipelines to use? QLP, SPOC -> each one is optimized for different purpose e.g., stellar variability, exoplanets,... Massive data set -> how to build an easy interface option suitable for ML?)
  • Gaia (Very sparse light curves compared to Kepler/TESS)

Spectra:

Questions to answer:

• How to easily cross-match all of them? Gaia IDs?
• What do do with raw full frame images? Light curves are essentially derived data products (see comment about pipeline)

[erinhay]

YSE DR1 Zenodo link: https://zenodo.org/records/7317476 PLAsTiCC (unbinded) Zenodo link: https://zenodo.org/records/2539456

[erinhay]

Added functionality to download YSE DR1 data from Zenodo

https://github.com/AstroPile/AstroPile_prototype/tree/yse

[helenqu]

time series format (zero-padded to max length):

• time: absolute time in MJD (float32)
• flux
• flux_err
• band: integer band ID
• quality_mask: @jeraud can maybe expand on these details

required metadata:

• object_id: unique within a survey/data source
• ra
• dec

optional metadata (e.g. host galaxy info) can be added for different surveys/sources

[helenqu]

plasticc: https://github.com/AstroPile/AstroPile_prototype/pull/22

[tom-hehir]

Have converted YSE data to HDF5 format (may still want to add some more optional metadata columns and rethink formatting of quality mask which is currently 0x00 for all data).

[helenqu]

time series row per object: (n_bands, n_features, seq_len)

• n_bands: number of bands
• n_features: likely 3 (timestamp in MJD, flux measurement, flux error)
• seq_len: length of the sequence (padded to the longest sequence of the file)

[mb010]

@helenqu the Cambridge group (@tom-hehir @ado8 @benboyd97 @David-Chemaly) is planning on writting a timedomain.py file that has a huggingface dataset defined in the morning. I think we are struggling to find a mental bridge between the data prep (which your PR does for plasticc) and the HF datasets format. So we are going to write a file to that end to have a target to build the data for. Let us know where you get up to today so we don't duplicate any of your work. 😄

[helenqu]

@mb010 thanks for letting me know! i just read this and was preparing a dataset loading script. i'm not done yet but will update the PR so you guys can see (will mark where i stopped).

[benboyd97]

I added the Foundation DR1 data_download.py to the yse branch. We'll create a pull request for the main branch once we've figured out some bugs with the hdf5 file storage.

[ado8]

I added some basic python files for downloading data and building parent samples for CSP DR3 and CfA's SNe II. There's a lot of copy-pasting so there might be room for consolidating common functions.

[benboyd97]

Added four more SNe Ia datasets from the Pantheon+ compilation, detailed further in issue #30. Work can be found in the panth_sne branch.

[helenqu]

finally have dataset loading working for plasticc!

updated code in https://github.com/AstroPile/AstroPile_prototype/pull/22

[tom-hehir]

Have updated build_parent_sample and verified that it works on the following datasets: YSE, SNLS, Foundation, DES Y3

However script currently breaks on Swift and PS1 with both giving the same error message:

Traceback (most recent call last):
  File "C:\Users\tomhe\Documents\AstroPile_prototype\scripts\ps1_sne_ia\build_parent_sample.py", line 161, in <module>
    main(args)
  File "C:\Users\tomhe\Documents\AstroPile_prototype\scripts\ps1_sne_ia\build_parent_sample.py", line 70, in main
    metadata[key].append(metadata_[key])
                         ~~~~~~~~~^^^^^
KeyError: 'SNTYPE'

Will investigate this tomorrow -- expect it will be a quick fix...

[tom-hehir]

Update on storage of timeseries data:

• The standard format requires us to store: time, flux, flux_err organised by band into a block of shape (num_examples, num_bands, 3, sequence_length)
• But most surveys will also have other timeseries variables e.g. mag and mag_err
• I am currently creating two arrays: lightcurve and lightcurve_additional and storing the core timeseries variables in the former and additional timeseries variables in the latter. Another option would be to store them in a single block (but this is in conflict with the suggestion below...)

Possible new convention for storage:

• When presenting data to the user, we do not collect the timeseries into a block of shape (num_examples, num_bands, 3, sequence_length), instead the user can specifically access e.g. the flux timeseries by indexing the data with [flux].
• I propose that we store the data similarly i.e. separate each timeseries into a different column of the HDF5 so that it is accessed individually by the corresponding key, rather than grouping all timeseries into a block (which must have a particular ordering to know which axis contains which data -- less straightforward / user friendly than key-based access).
• Why? Although the core timeseries variables (time, flux, flux_err) can all be represented as np.float32 and put into a single array, other timeseries have heterogeneous data types which prohibits combining them into a single array.

[mb010]

Mini summary of our discussion:

TODOs:

• [ ] add more surveys (also look at Matt's image data above) @everyone
• [ ] look how to read @ado8 's non-SNANA filetype data (csp branch) and save it in hdf5 format
• [x] find healpix ID for each SN and save hdf5 files accordingly @benboyd97 @tom-hehir
• [ ] change data loader code to expect separate hdf5 files for each healpix ID @mb010 @helenqu
• [ ] add citation data etc. for other surveys besides yse into data loaders @everyone
• [ ] write test.sh script for each dataset
• [ ] @jeraud checking that his HF format matches the basic features of the (now) template format in the yse.py file