Ingest and publish LIS DA simulation datasets

[slesaad]

Description

To support discoveries being developed by EIS scientists, there have been requests for dataset publication:

This ticket includes ingestion/publication of 11 datasets:

• [x] ~ 20 years of daily output for 9 different variables from LIS DA simulation (dev ✅ , staging ❌ )
  • [x] lisDA_Evap
  • [x] lisDA_GPP
  • [x] lisDA_GWS
  • [x] lisDA_Qs
  • [x] lisDA_Qsb
  • [x] lisDA_Streamflow
  • [x] lisDA_SWE
  • [x] lisDA_TWS
  • [x] lisDA_TotalPrecip
• [x] 2 trend datasets over ~20 years (dev ✅ , staging ❌ )
  • [x] lis-da-tws-trend
  • [x] lis-da-gpp-trend

More details in this slack thread: https://teamdsig.slack.com/archives/C04A33BUY8J/p1676390322600719

[slesaad]

The 2 trend datasets have been published:

1. https://staging-stac.delta-backend.com/collections/lis-global-da-gpp-trend
2. https://staging-stac.delta-backend.com/collections/lis-tws-trend

Following inputs used:

{
    "collection": "lis-global-da-gpp-trend",
    "title": "Gross Primary Productivity Trend - LIS 10km Global DA",
    "description": "Gridded trend in gross primary productivity (theil-sen slope estimation in gC m-2 yr-1) from 10km global LIS with assimilation",
    "license": "CC0-1.0",
    "is_periodic": false,
    "time_density": null,
    "spatial_extent": {
      "xmin": -179.95,
      "ymin": -59.45,
      "xmax": 179.95,
      "ymax": 83.55
    },
    "temporal_extent": {
      "startdate": "2003-01-01T00:00:00Z",
      "enddate": "2021-12-31T23:59:59Z"
    },
    "sample_files": [
      "s3://veda-data-store-staging/EIS/COG/LIS_GLOBAL_DA/DA_Trends/DAGPP_STL_based_trend.cog.tif"
    ],
    "discovery_items": [
      {
        "discovery": "s3",
        "cogify": false,
        "upload": false,
        "dry_run": false,
        "prefix": "EIS/COG/LIS_GLOBAL_DA/DA_Trends/",
        "bucket": "veda-data-store-staging",
        "filename_regex": "(.*)DAGPP_STL_based_trend.cog.tif$",
        "start_datetime": "2003-01-01T00:00:00Z",
        "end_datetime": "2021-12-31T23:59:59Z"
      }
    ]
  }

{
  "collection": "lis-global-da-tws-trend",
  "title": "Terrestrial Water Storage Trend - LIS 10km Global DA",
  "description": "Gridded trend in terrestrial water storage (theil-sen slope estimation in mm yr-1) from 10km global LIS with assimilation",
  "license": "CC0-1.0",
  "is_periodic": false,
  "time_density": null,
  "spatial_extent": {
    "xmin": -179.95,
    "ymin": -59.45,
    "xmax": 179.95,
    "ymax": 83.55
  },
  "temporal_extent": {
    "startdate": "2003-01-01T00:00:00Z",
    "enddate": "2021-12-31T23:59:59Z"
  },
  "sample_files": [
    "s3://veda-data-store-staging/EIS/COG/LIS_GLOBAL_DA/DA_Trends/DATWS_STL_based_trend.cog.tif"
  ],
  "discovery_items": [
    {
      "discovery": "s3",
      "cogify": false,
      "upload": false,
      "dry_run": false,
      "prefix": "EIS/COG/LIS_GLOBAL_DA/DA_Trends/",
      "bucket": "veda-data-store-staging",
      "filename_regex": "(.*)DATWS_STL_based_trend.cog.tif$",
      "start_datetime": "2003-01-01T00:00:00Z",
      "end_datetime": "2021-12-31T23:59:59Z"
    }
  ]
}

[slesaad]

the datasets have been published to dev catalog, waiting on review to publish to staging

[slesaad]

waiting on staging airflow to work to ingest these datasets:

the dataset definitions are in this PR: https://github.com/NASA-IMPACT/veda-data-pipelines/pull/311 the collections are published already, but since I couldn't run the workflows, the items are empty

@moradology could you run these datasets via staging airflow once the issue is fixed? the definitions are in "discovery_items" section of the jsons, will need to add collection to them. thanks!

[moradology]

Staging ingests successfully completed!

[slesaad]

thanks @moradology ! 🙇