search

NASA-IMPACT / veda-data

2 stars 0 forks source link

Rename yearly NO2 collection from OMI_trno2-COG to no2-yearly #141

Open j08lue opened 3 weeks ago

j08lue commented 3 weeks ago

We currently have three NO2 products in our catalog:

They present the same type of property (NO2 concentrations), just different derived products. Would it make sense to rename the OMI_trno2-COG collection to no2-yearly or so, to harmonize the naming pattern between the three?

We would need to update it in VEDA Earthdata Dashboard and the EO Dashboard, but I think that is easy enough to do.

Acceptance criteria

anayeaye commented 3 weeks ago

@j08lue This change of collection id is better handled by creating a new collection because the id is an index and the database and the collection id maps 1:1 to the s3 prefix in the production data store. If we decide to do this, et's instead:

  1. duplicate all object in veda-data-store/OMI_trno2-COGto veda-data-store/no2-yearly
  2. (a) create a new ingest-items/production/collection with id no2-yearly.json here in veda-data along with a (b) discovery items config file (it should be easy to produce following how the original collection and discovery were configured. (c) We will also need a transfer-config json that shows how the UAH covid-eo-dashboard/OMI_trno2-COG collection maps to no2-yearly in production (I believe this is an ongoing collection?)
  3. publish the new collection and ingest all items
  4. announce that the old collection_id is deprecated and coordinate with the dashboard team, EOX, and EIC
  5. finally when all three dashboards are updated for the new collection_id we can begin the deprecation process of deleting the stac records and setting a lifecycle policy on the deprecated prefix in s3
j08lue commented 2 weeks ago

Great to see the process laid out this clearly, @anayeaye, and sounds like a very solid approach. Btw, we should copy this to operations docs somewhere - like "How to rename collections" and "How to deprecate collections"!

Not sure it is worth the effort for the OMI rename, though? What do others think - @anayeaye, @slesaad, @smohiudd?

anayeaye commented 2 weeks ago

Btw, we should copy this to operations docs somewhere - like "How to rename collections" and "How to deprecate collections"!

Yes! We do also have some work planned on covering how to remove VEDA data safely #116, #117 Maybe we could sneak in some differentiation on collection id vs name in that documentation, too? It is easy to change the collection name but that index is a more significant change. On the same note, maybe we should add waaaaay more emphasis on choosing a collection id in our contributing docs https://nasa-impact.github.io/veda-docs/contributing/dataset-ingestion/. I think it is not initially clear how valuable a human readable meaningful STAC Collection id is.