Distribute UCoMP tarballs/files via website - Githubissues

NCAR / ucomp-pipeline

Data processing pipeline for UCoMP

Other

6 stars 3 forks source link

Distribute UCoMP tarballs/files via website #204

Closed bberkeyU closed 1 year ago

bberkeyU commented 1 year ago

Individual files

This includes all .png, .gif, and .mp4 quicklook files corresponding to level 1 and 2 FITS files.

level 1 mp4s and all level 2 quicklooks to fullres date directory
(later) level 2 FITS files to archive date directory (this will be needed for data delivery via web service)

Tarballs

This includes all tarballs and zip files collecting data, metadata, or quicklook files.

Data listing for the day zip

filename YYYYMMDD.ucomp.catalogs.zip in fullres
list filename YYYYMMDD.ucomp.catalogs.ziplist.txt in fullres
Users Guide (short description with link to website for now)
catalog files (all files and files by wave region)
quality files
GBU files
Data citation info

Quicklook gzip (all wavelengths)

filename YYYYMMDD.ucomp.quicklooks.zip in fullres
filename YYYYMMDD.ucomp.quicklooks.ziplist.txt in fullres
all L1/L2 quicklook PNG, GIF, and movies
Users Guide (short description with link to website for now)
catalog files (all files and files by wave region)
quality files
GBU files
Data Citation info

All level 1 FITS (all wavelengths)

filename YYYYMMDD.ucomp.l1.tar.gz in archive
list filename YYYYMMDD.ucomp.l1.tarlist.txt in archive
mean/median average (waves and synoptic) L1-like files
individual L1 files that pass L0 quality check
Users Guide (short description with link to website for now)
catalog files (all files and files by wave region)
quality files
GBU files
Data Citation info

All level 2 FITS (all wavelengths)

filename YYYYMMDD.ucomp.l2.tar.gz in archive
filename YYYYMMDD.ucomp.l2.tarlist.txt in archive
mean/median waves and synoptic L2 files
individual L2 files files
Users Guide (short description with link to website for now)
catalog files
quality files
GBU files
Data Citation info

Level 1 and level 2 mean/median waves/synoptic only (all wavelengths)

filename YYYYMMDD.ucomp.daily_averages.tar.gz in archive
filename YYYYMMDD.ucomp.daily_averages.tarlist.txt in archive
mean/median waves and synoptic L1 and L2 files
sigmas files? depending on #167
Users Guide (short description with link to website for now)
catalog files (all files and files by wave region)
quality files
GBU files
Data citation info

Tasks

[x] YYYYMMDD.ucomp.catalogs.zip
[x] YYYYMMDD.ucomp.quicklooks.zip
[x] YYYYMMDD.ucomp.l1.tar.gz
[x] YYYYMMDD.ucomp.l2.tar.gz
[x] YYYYMMDD.ucomp.daily_averages.tar.gz

mgalloy commented 1 year ago

Be aware there is a YYYYMMDD.ucomp.catalog.txt (a catalog of all the level 0 UCoMP files for the day) and a YYYYMMDD.ucomp.catalogs.txt (a listing of all the files in YYYYMMDD.ucomp.catalog.zip) in the fullres directory.