Data transfer to MCP

[slesaad]

Epic

118

Description

One of the most important tasks in migration of VEDA to MCP is migration of the data files to the MCP s3 bucket.

Based on decisions made in #118, it's been decided that this will be carried out by running the data transformation and ingestion pipeline with a flag set to transfer the data to the MCP s3 bucket.

A role based policy will be used to gain access to the MCP s3 bucket from the ingestion pipeline.

Examples

The current data link points to a UAH bucket. That's where the data files exist.

{
  "assets": {
    "cog_default": {
      "href": "s3://climatedashboard-data/bmhd_30m_monthly_bkp/finalBMHD_ScaledVenice_202203.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "roles": [
        "data",
        "layer"
      ]
    },
  }
}

At the end of the data migration, the links should look like the following and the data should exist in that link:

{
  "assets": {
    "cog_default": {
      "href": "s3://veda-data-store-staging/nightlights-hd-monthly/finalBMHD_ScaledVenice_202203.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "roles": [
        "data",
        "layer"
      ]
    },
  }
}

Acceptance Criteria:

• All the data is transferred to the MCP bucket
• All the data link points to the MCP s3 url
• The asset/data links from the staging catalog can still be viewed in the cog viewer after the transfer

Checklist for collections

• [x] nightlights-hd-1band
• [x] social-vulnerability-index-housing-nopop
• [x] grdi-v1-built
• [x] nightlights-hd-monthly
• [x] MO_NPP_npp_vgpm
• HLSS30.002 (these are in lpdaac protected buckets; @anayeaye is working on creating our own version of these)
• HLSL30.002 ( same )
• [x] OMI_trno2-COG
• [x] social-vulnerability-index-household
• [x] grdi-v1-raster
• [x] OMSO2PCA-COG
• [x] grdi-shdi-raster
• [x] facebook_population_density
• [x] grdi-vnl-slope-raster
• [x] social-vulnerability-index-socioeconomic
• [x] social-vulnerability-index-socioeconomic-nopop
• [x] grdi-filled-missing-values-count
• [x] grdi-vnl-raster
• [x] grdi-cdr-raster
• [x] blue-tarp-planetscope
• [x] no2-monthly-diff
• [x] social-vulnerability-index-household-nopop
• [x] social-vulnerability-index-minority
• [x] social-vulnerability-index-overall-nopop
• [x] no2-monthly
• [x] social-vulnerability-index-overall
• [x] social-vulnerability-index-housing
• [x] blue-tarp-detection
• [x] IS2SITMOGR4-cog
• [x] social-vulnerability-index-minority-nopop
• [x] nceo_africa_2017
• [x] geoglam
• [x] grdi-imr-raster
• nightlights-hd-3bands

Checklist:

• [x] Epic Link
• [x] Detailed description
• Concept diagrams
• [x] Assignee

[xhagrg]

We should probably push to prod bucket rather than staging bucket.

[slesaad]

Misc TODOs

• [x] Remove povmap-grdi-v1_VNL-2020-01-01_2020-12-31 from grdi-vnl-raster
• [x] Delete collection IS2SITMOGR4, added IS2SITMOGR4-cog instead
• [x] Delete nightlights-hd-3bands
• [x] Add nightlights-500m-daily

[slesaad]

Some pgstac database quirks ☠️ realised while migrating the datasets that we should be aware of:

• When you update a collection after the items have been ingested into the collection, the items disappear. 🛑 DONOT UPDATE A COLLECTION AFTER THE ITEMS ARE INGESTED, else you'll have to run the ingestion again.
• We can't use same id 👯 for items even if they are in different collection

[abarciauskas-bgse]

The data products are in s3://veda-data-store-staging and not s3://veda-data-store - @xhagrg @slesaad can we migrate the products to be in s3://veda-data-store? 🙏🏽

[abarciauskas-bgse]

We are having a longer discussion on slack so will follow up with next steps