search

hytest-org / hytest

https://hytest-org.github.io/hytest/
22 stars 10 forks source link

Remove calculated variables from CONUS404 zarrs #386

Open amsnyder opened 9 months ago

amsnyder commented 9 months ago

Variables to remove: E2, ES2, RH2, SH2 Based on: https://doimspp.sharepoint.com/:w:/r/sites/gs-wma-hytest/_layouts/15/Doc.aspx?sourcedoc=%7B49275EF5-7120-4B12-A3B3-94782ECC8AE0%7D&file=CONUS404%20Zarr%20Derived%20Variables.docx&action=default&mobileredirect=true

For now, remove from metadata so they don't show up, but keep data in zarr store.

amsnyder commented 8 months ago

Will keep the derived variables PWAT, TK, Z because they are published on RDA. They are only available in the 3D files, so it would be difficult to reproduce if needed.

amsnyder commented 8 months ago

@pnorton-usgs will edit zmetadata file on Caldera and write some notes about what was removed and how it can be added back to restore functionality of variables.

Amelia will sync this change to S3 and OSN

pnorton-usgs commented 8 months ago

@amsnyder The zmetadata files for the conus404-raw hourly, daily, and monthly zarr stores have been updated and verified for the onprem datasets.

amsnyder commented 8 months ago

The updated metadata has been copied onto S3 and the OSN pod.

pnorton-usgs commented 5 months ago

@amsnyder Recently the conus404-raw hourly, daily, and monthly zarr stores were extended to include time steps through 2022-10-01 (see #369). The derived variables E2, ES2, RH2, and SH2 were not updated in the hourly zarr and the daily and monthly zarr stores were recreated so they no longer have those derived variables at all. I've reopened this issue to keep continuity with our prior decision about keeping the derived variables but removing them from the consolidated metadata. I propose we remove these derived variables entirely from the hourly zarr store. If they are needed at some point in the future we can recreate these variables with our workflows.

pnorton-usgs commented 5 months ago

@amsnyder I have removed E2, ES2, RH2, and SH2 entirely from the conus404-raw hourly on-prem dataset. Once these variables are also removed from our cloud holdings this issue can be closed (again).