[Data] JWST / HST deep surveys

[mhuertascompany]

JWST / HST deep surveys

Include a new dataset in astropile from public deep HST and JWST galaxy surveys

Contacts: Participants: Marc Huertas-Company + anyone interested

Goals and deliverable

Building an astropile dataset with JWST and HST (imaging) stamps. Investigate if spectra for some sources can be added. The main source of data is MAST. However I expect that catalog information is in different formats for different datasets. Homogenization might be an issue.

Resources needed

mostly enthusiasm + familiarity with astropile format + imaging / MAST archives.

Detailed description

[add additional details about the project]

[mhuertascompany]

started to download COSMOS-Web data from DJA. created branch jwst. working for now.

[mhuertascompany]

Worked on downloading CEERS data from DJA. COSMOS-Web data is still not complete. CEERS contains ~ 70.000 galaxies. we have now a running script that downloads large field images, cuts stamps and stores them in a hdf5 file, following the same structure as for the hsc dataset. all updates pushed to jwst branch. for now it's in notebook form..

ToDo: convert to astropile format - looking at this function: def save_in_standard_format(catalog_filename, cutouts_filename, output_dir, num_processes=None):

[mhuertascompany]

JWST HF dataset working Tested on Liam's data loader.

[mhuertascompany]

Run a photoz baseline network using 6 band imaging + Liam's wrapper. Disaster.

[kartheikiyer]

this is similar to what i got yesterday with the IOB+ILI, so it might be more of a dataset issue than a pipeline/dataloader issue. its good to know that its reproducible. .)

[mhuertascompany]

trying a flow on the summary statistics of the resnet...still training.

[kartheikiyer]

Finally managed to get the data preprocessing + loading classes working across multiple fields. Made a basic comparison of wide / deep / cluster fields to see how things look.

Plot 1: comparison of different fields imported from DjA in the same format. needed to manually add a couple of photoz files that weren't in the correct format. But compatible with the hdf5 loader @mhuertascompany wrote and the training dataset wrapper from @lhparker1 .

Couple more plots incoming (and code to be added to the repo soon).

[kartheikiyer]

trained a basic AE with the IOB layer (paper) using only single band (F200w) images for now, but the code can be scaled to arbitrary filtersets.

[kartheikiyer]

The latent representation learned can then be compressed and correlated with properties (and we can compare which fields lie in different parts of the latent space). Plot here using PaCMap.

Plot: redshift correlates with location in latent space, which makes me think we ought to be seeing a better redshift prediction, or something else is going on. (e.g. the IOB representation is learning SNR/compactness).

Plot: latent space coded by which field contributes the maximum galaxies in any cell. probably wrong/needs to be renormed for sample size.