Add data from Copernicus products

[dotis]

Example CMEMS dataset (Global carbon): https://data.marine.copernicus.eu/product/MULTIOBS_GLO_BIO_CARBON_SURFACE_REP_015_008/description

Data access page is here: https://data.marine.copernicus.eu/product/MULTIOBS_GLO_BIO_CARBON_SURFACE_REP_015_008/services

No ERDDAP option. Other options for data download: MOTU (can subset) OpenDAP FTP WMS (maps)

Need login credentials for this. Account is free (CMEMS).

MOTU API URL: python -m motuclient --motu https://my.cmems-du.eu/motu-web/Motu --service-id MULTIOBS_GLO_BIO_CARBON_SURFACE_REP_015_008-TDS --product-id dataset-carbon-rep-monthly --date-min "2021-12-15 00:00:00" --date-max "2021-12-15 23:59:59" --variable fgco2 --variable fgco2_uncertainty --variable omega_ar --variable omega_ar_uncertainty --variable omega_ca --variable omega_ca_uncertainty --variable ph --variable ph_uncertainty --variable spco2 --variable spco2_uncertainty --variable talk --variable talk_uncertainty --variable tco2 --variable tco2_uncertainty --out-dir --out-name --user --pwd

[7yl4r]

The motu API python client looks like a good option. Instead of passing all the params as arguments, the --config-file parameter will allow us to put these (and login credentials) into a file. [ref] We can create a file for that here, but keep the username and password parameters out of the file. For those we want to set up "secrets" in the github configuration so that we can use them in gh-actions.

• [ ] config file(s) create
• [ ] username & password secrets created
• [ ] copernicus-download gh-action ingest workflow created
• [ ] gh-action is working
• [ ] data from the gh-action added to dashboard view

[7yl4r]

Subsetting the datasets using MOTU is going to be a challenge very similar to the ones we have with ERDDAP. MOTU can subset using a bounding box and time. I don't know what file format you get from a query like this:

#!/bin/bash
# work in progress download data from motu
python -m motuclient --motu https://my.cmems-du.eu/motu-web/Motu \
  --service-id MULTIOBS_GLO_BIO_CARBON_SURFACE_REP_015_008-TDS \
  --product-id dataset-carbon-rep-monthly \
  --date-min "2021-12-15 00:00:00" --date-max "2021-12-15 23:59:59" \
  --variable fgco2 --variable fgco2_uncertainty --variable omega_ar \
  --variable omega_ar_uncertainty --variable omega_ca \
  --variable omega_ca_uncertainty --variable ph \
  --variable ph_uncertainty --variable spco2 \
  --variable spco2_uncertainty --variable talk \
  --variable talk_uncertainty --variable tco2 \
  --variable tco2_uncertainty \
  --out-dir <OUTPUT_DIRECTORY> \
  --out-name <OUTPUT_FILENAME> \
  --user --pwd 

It would be lovely if ERDDAP could be set up to connect to a MOTU backend (like it does with other ERDDAP servers & s3 buckets), but I don't see that in the ERDDAP docs. A PR to ERDDAP would be needed.

The alternatives:

• do the fetching and subsetting here
• set up something to do an automated download of files from MOTU into ERDDAP

[7yl4r]

closing here and moving discussion over to https://github.com/USF-IMARS/erddap-config/issues/52