Earth Data Collection Requirements

tinaok commented 5 months ago

Objective

The goal of this issue is to outline and track the collection of essential earth data products necessary for our project. This table will serve as a reference to ensure we have all the necessary data ready and accessible for project phases requiring them.

Data Requirements

We have two main data categories for our project:

Past Data: Required for quick-track construction from biologging data sets.
Future Data: Needed to estimate future habitat environmental stress.

Please find below the table detailing each data product needed for the project. Each row represents a specific data type, with details about its purpose, source, deadline, and preferred format.

Past Data:

Sea Temperature, Sea surface hight, hourly and Daily

**CMEMS:***

CMEMS Data Product Name	spatial extent and resolution	Temporal extent	Source(DOI)	Format	Availability	DEDL
Atlantic: Iberia-Biscay-IrelandAtlantic: North IBI_ANALYSISFORECAST_PHY_005_001	Lat 26° to 56°Lon -19° to 5°, 0.083 deg	1 Dec 2020 to now	https://doi.org/10.48670/moi-00027	NetCDF & ZARR (if zarr , chunked spatially)	-	-
Atlantic: IBI_MULTIYEAR_PHY_005_002	Lat 26° to 56°Lon -19° to 5°, 0.028° × 0.028°	1 Dec 2020 to now	https://doi.org/10.48670/moi-00027	NetCDF & ZARR (if zarr , chunked spatially)	-	-
Atlantic: European North West Shelf- Ocean Physics Reanalysis: NWSHELF_ANALYSISFORECAST_PHY_004_013	Lat 46° to 61.28°Lon -16° to 9.98°,0.014° × 0.03°	1 Sep 2021 to now	https://doi.org/10.48670/moi-00054, decommissioned from CMEMS, see https://www.metoffice.gov.uk/services/data/met-office-marine-data-service	NetCDF & ZARR (if zarr , chunked spatially)	-	-
NWSHELF_ANALYSISFORECAST_PHY_004_009	Lat 46° to 61.28°Lon -16° to 9.98°,0.014° × 0.03°			NetCDF & ZARR (if zarr , chunked spatially)	-	-
GLOBAL_ANALYSISFORECAST_PHY_001_024	Global	2020 to now. All the version of data	7km resolution	NetCDF & ZARR (if zarr , chunked spatially)	-	-
GLOBAL_MULTIYEAR_PHY_001_030	Global	two product. one untill 2021, another from 2021, separated in 2 zarr file		NetCDF & ZARR (if zarr , chunked spatially)	-	-

| ... | ... | ... | ... | ... | ... |

Future Data:

(https://easy.gems.dkrz.de/DYAMOND/NextGEMS/cycle3.html#nextgems-cycle-3 for the model description?)	Data Category	Data Product	Description	Source(DOI)	Format	Availability
FESOM (70 levels)	Temperature, SSH, Salinity,...	?reference paper?	Grib by polytope	transform to zarr or kechunk of Grib	?	?
NEMO (75 levels)	Temperature, SSH, Salinity,...	?reference paper?	Grib by polytope	transform to zarr or kechunk of Grib	?	?
ICON (? levels)	Temperature, SSH, Salinity,...	?reference paper?	Grib by polytope	transform to zarr or kechunk of Grib	?	?
RIOMAR (? levels)	Temperature, SSH, Salinity,...	?reference paper?	NetCDF, stored on HPC center.	transform to zarr or kechunk of Grib	?	?

Data Management and Tracking

Each data product can have a sub-issue to track its availability:

Is the data available on DEDL (Data Environment for Data Libraries)?
If not, the request to DEDL is made?
While waiting for availability on DEDL, should we make a copy to our DESP (Data Environment for Special Projects) space?
Status updates for copying data, e.g., "Copied in Zarr format?"
If in GRIB/NetCDF, is Kerchunk preparation completed?
Is the Intake catalog prepared for accessing the data?

Request for Contribution

I encourage team members to contribute by identifying potential data sources or by assisting in the data collection process. Please update this table or comment on this issue as you make progress or if you encounter any challenges.

Additional Notes

Ensure all collected data complies with legal and privacy requirements.
Regularly update the status of data preparation and accessibility.

Thank you for your contributions and let's make sure all necessary data is collected promptly and efficiently!

annefou commented 5 months ago

For the ClimateDT, I managed to get one month (June as requested by Tina) for ifs-nemo (I also tried icon but I can't get much; not sure why):

160366332 avg_hc300m_ifs-nemo_20210601-20210615.grib
160590884 avg_hc300m_ifs-nemo_20210616-20210630.grib
175705501 avg_hc300m_ifs-nemo_20220601-20220615.grib
171527445 avg_hc300m_ifs-nemo_20220616-20220630.grib
160499497 avg_hc300m_ifs-nemo_20230601-20230615.grib
165171522 avg_hc300m_ifs-nemo_20230616-20230630.grib
160620926 avg_hc300m_ifs-nemo_20240601-20240615.grib
160756733 avg_hc300m_ifs-nemo_20240616-20240630.grib
154134651 avg_hc700m_ifs-nemo_20210601-20210615.grib
154250888 avg_hc700m_ifs-nemo_20210616-20210630.grib
153740121 avg_hc700m_ifs-nemo_20220601-20220615.grib
153932408 avg_hc700m_ifs-nemo_20220616-20220630.grib
154003557 avg_hc700m_ifs-nemo_20230601-20230615.grib
154146803 avg_hc700m_ifs-nemo_20230616-20230630.grib
153969132 avg_hc700m_ifs-nemo_20240601-20240615.grib
154056305 avg_hc700m_ifs-nemo_20240616-20240630.grib
6495121699 avg_so_ifs-nemo_20210601-20210615.grib
6494203690 avg_so_ifs-nemo_20210616-20210630.grib
6497145446 avg_so_ifs-nemo_20220601-20220615.grib
6498916071 avg_so_ifs-nemo_20220616-20220630.grib
6477147301 avg_so_ifs-nemo_20230601-20230615.grib
6478819417 avg_so_ifs-nemo_20230616-20230630.grib
6492560276 avg_so_ifs-nemo_20240601-20240615.grib
6493759416 avg_so_ifs-nemo_20240616-20240630.grib
124430145 avg_sos_ifs-nemo_20210601-20210615.grib
124504649 avg_sos_ifs-nemo_20210616-20210630.grib
125003963 avg_sos_ifs-nemo_20220601-20220615.grib
124958912 avg_sos_ifs-nemo_20220616-20220630.grib
124773729 avg_sos_ifs-nemo_20230601-20230615.grib
124706330 avg_sos_ifs-nemo_20230616-20230630.grib
125208677 avg_sos_ifs-nemo_20240601-20240615.grib
125210521 avg_sos_ifs-nemo_20240616-20240630.grib
9959434512 avg_thetao_icon_20210601-20210615.grib
9956652167 avg_thetao_icon_20210616-20210630.grib
9961834898 avg_thetao_icon_20220601-20220615.grib
9234391819 avg_thetao_ifs-nemo_20210601-20210615.grib
9231699251 avg_thetao_ifs-nemo_20210616-20210630.grib
9242519770 avg_thetao_ifs-nemo_20220601-20220615.grib
9242566457 avg_thetao_ifs-nemo_20220616-20220630.grib
9234889485 avg_thetao_ifs-nemo_20230601-20230615.grib
9235449623 avg_thetao_ifs-nemo_20230616-20230630.grib
9257534960 avg_thetao_ifs-nemo_20240601-20240615.grib
9256437221 avg_thetao_ifs-nemo_20240616-20240630.grib
152915639 avg_tos_ifs-nemo_20210601-20210615.grib
152917464 avg_tos_ifs-nemo_20210616-20210630.grib
153419426 avg_tos_ifs-nemo_20220601-20220615.grib
153259330 avg_tos_ifs-nemo_20220616-20220630.grib
152716339 avg_tos_ifs-nemo_20230601-20230615.grib
152801249 avg_tos_ifs-nemo_20230616-20230630.grib
153594860 avg_tos_ifs-nemo_20240601-20240615.grib
153345960 avg_tos_ifs-nemo_20240616-20240630.grib
11080412455 avg_uoe_ifs-nemo_20210601-20210615.grib
10889924371 avg_uoe_ifs-nemo_20210616-20210630.grib
10929602005 avg_uoe_ifs-nemo_20220601-20220615.grib
10865601865 avg_uoe_ifs-nemo_20220616-20220630.grib
10977099930 avg_uoe_ifs-nemo_20230601-20230615.grib
10950679268 avg_uoe_ifs-nemo_20230616-20230630.grib
10965176810 avg_uoe_ifs-nemo_20240601-20240615.grib
10915231921 avg_uoe_ifs-nemo_20240616-20240630.grib
11126417501 avg_von_ifs-nemo_20210601-20210615.grib
11025381071 avg_von_ifs-nemo_20210616-20210630.grib
11069287606 avg_von_ifs-nemo_20220601-20220615.grib
10921799480 avg_von_ifs-nemo_20220616-20220630.grib
11132575892 avg_von_ifs-nemo_20230601-20230615.grib
11025466889 avg_von_ifs-nemo_20230616-20230630.grib
11230409480 avg_von_ifs-nemo_20240601-20240615.grib
11123705070 avg_von_ifs-nemo_20240616-20240630.grib
9169116191 avg_wo_ifs-nemo_20210601-20210615.grib
9245560393 avg_wo_ifs-nemo_20210616-20210630.grib
9314574859 avg_wo_ifs-nemo_20220601-20220615.grib
9276597759 avg_wo_ifs-nemo_20220616-20220630.grib
9332851551 avg_wo_ifs-nemo_20230601-20230615.grib
9391269058 avg_wo_ifs-nemo_20230616-20230630.grib
9290668141 avg_wo_ifs-nemo_20240601-20240615.grib
9478879455 avg_wo_ifs-nemo_20240616-20240630.grib
135325122 avg_zos_icon_20240616-20240630.grib
159751128 avg_zos_ifs-nemo_20210601-20210615.grib
159893845 avg_zos_ifs-nemo_20210616-20210630.grib
159575287 avg_zos_ifs-nemo_20220601-20220615.grib
159600267 avg_zos_ifs-nemo_20220616-20220630.grib
159870669 avg_zos_ifs-nemo_20230601-20230615.grib
160013276 avg_zos_ifs-nemo_20230616-20230630.grib
159815137 avg_zos_ifs-nemo_20240601-20240615.grib
159794729 avg_zos_ifs-nemo_20240616-20240630.grib
  6963826 lsm.grib
215529769 slhf_icon_20210601-20210615.grib
211385292 slhf_icon_20210616-20210630.grib
218094425 slhf_icon_20220601-20220615.grib
215816467 slhf_icon_20220616-20220630.grib
222054855 slhf_icon_20230601-20230615.grib
222463963 slhf_icon_20230616-20230630.grib
215814211 slhf_icon_20240601-20240615.grib
220679898 slhf_icon_20240616-20240630.grib
413306346 slhf_ifs-nemo_20210601-20210615.grib
412586498 slhf_ifs-nemo_20210616-20210630.grib
406136551 slhf_ifs-nemo_20220601-20220615.grib
402732836 slhf_ifs-nemo_20220616-20220630.grib
413505983 slhf_ifs-nemo_20230601-20230615.grib
410592215 slhf_ifs-nemo_20230616-20230630.grib
411291551 slhf_ifs-nemo_20240601-20240615.grib
406442753 slhf_ifs-nemo_20240616-20240630.grib
199254606 sshf_icon_20210601-20210615.grib
186454593 sshf_icon_20220601-20220615.grib
196349123 sshf_icon_20220616-20220630.grib
199066550 sshf_icon_20230601-20230615.grib
199342620 sshf_icon_20230616-20230630.grib
200491041 sshf_icon_20240601-20240615.grib
200357028 sshf_icon_20240616-20240630.grib
395691070 sshf_ifs-nemo_20210601-20210615.grib
399767163 sshf_ifs-nemo_20210616-20210630.grib
395065383 sshf_ifs-nemo_20220601-20220615.grib
394989618 sshf_ifs-nemo_20220616-20220630.grib
396834844 sshf_ifs-nemo_20230601-20230615.grib
396999891 sshf_ifs-nemo_20230616-20230630.grib
399883620 sshf_ifs-nemo_20240601-20240615.grib
398481366 sshf_ifs-nemo_20240616-20240630.grib

@tinaok would you need other months, etc?

tinaok commented 5 months ago

Thanks @annefou, I would like to have them in kerchunk if possible.

annefou commented 5 months ago

Thanks @annefou, I would like to have them in kerchunk if possible.

Yes, I will create Kerchunk catalogs for the ClimateDT data too but as we only have access to the data until May 17th, I want to make sure we get all we need.

tinaok commented 5 months ago

as we only have access to the data until May 17th, I want to make sure we get all we need.

OK, then having thetao and zos for all available data (from the oldest (2010?) to all available future) can be great.

annefou commented 5 months ago

OK, then having thetao and zos for all available data (from the oldest (2010?) to all available future) can be great.

there is no data before 2020 available from polytope. (and at the moment, we can only get ifs-demo).

tinaok commented 2 months ago

I listed product names from Copernicus Marine services that for our interest here. I'll add it in the description of issue. Not all has 3d temperature available for full time.

 name={'GLOBAL_ANALYSISFORECAST_PHY_001_024':
      {'thetao':
      {'H':'cmems_mod_glo_phy-thetao_anfc_0.083deg_PT6H-i_202406',
      'D':'cmems_mod_glo_phy-thetao_anfc_0.083deg_P1D-m_202406' },
      'zos':
       {'H':'cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406' ,
      'D':'cmems_mod_glo_phy_anfc_0.083deg_P1D-m_202406'},
       'deptho': 'cmems_mod_glo_phy_anfc_0.083deg_static_202211--ext--bathy'},
    'GLOBAL_MULTIYEAR_PHY_001_030':
      {'thetao':
      {'NEW':'cmems_mod_glo_phy_myint_0.083deg_P1D-m_202311',
      'OLD':'cmems_mod_glo_phy_my_0.083deg_P1D-m_202311' },
      'zos':
       {'NEW':'cmems_mod_glo_phy_myint_0.083deg_P1D-m_202311' ,
      'OLD':'cmems_mod_glo_phy_anfc_0.083deg_P1D-m_202406'},
       'deptho': 'cmems_mod_glo_phy_anfc_0.083deg_static_202211--ext--bathy'},
          'IBI_MULTIYEAR_PHY_005_002':
      {'thetao':
      {'H':'cmems_mod_ibi_phy_anfc_0.027deg-3D_PT1H-m_202211',
      'D':'cmems_mod_ibi_phy_anfc_0.027deg-3D_P1D-m_202211' },
      'zos':
       {'H':'cmems_mod_ibi_phy_anfc_0.027deg-2D_PT1H-m_202211' ,
      'D':'cmems_mod_ibi_phy_anfc_0.027deg-3D_P1D-m_202211'},
       'deptho': 'cmems_mod_ibi_phy_anfc_0.027deg-3D_static_202211--ext--bathy'},
                'IBI_ANALYSISFORECAST_PHY_005_001':
      {'thetao':
      {'H':'',
      'D':'cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m_202012' },
      'zos':
       {'H':'cmems_mod_ibi_phy_my_0.083deg-2D_PT1H-m_202012' ,
      'D':'cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m_202012'},
       'deptho': 'cmems_mod_ibi_phy_my_0.083deg-3D_static_202012--ext--bathy'},
                'NWSHELF_ANALYSISFORECAST_PHY_004_013':
      {'thetao':
      {'H':'cmems_mod_nws_phy_anfc_0.027deg-3D_PT1H-m_202309',
      'D':'cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_202309' },
      'zos':
       {'H':'cmems_mod_nws_phy_anfc_0.027deg-2D_PT1H-m_202309' ,
      'D':'cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_202309'},
       'deptho': 'cmems_mod_nws_phy_anfc_0.027deg-3D_static_202309--ext--bathy'},
          'NWSHELF_MULTIYEAR_PHY_004_009':
      {'thetao':
      {'H':'',
      'D':'cmems_mod_nws_phy-t_my_7km-3D_P1D-m_202012' },
      'zos':
       {'H':'' ,
      'D':'cmems_mod_nws_phy-ssh_my_7km-2D_P1D-m_202012'},
       'deptho': 'cmems_mod_nws_phy-bottomt_my_7km-2D_P1D-m_202012'},
         }

tinaok commented 2 months ago

We need a way to query time-range(of tag) and bbox( of simulation) to all available 'thetao' and 'zos' to query the available dataset. unfortunately the catalogue from cmems gives back the time-range for 'all' the catalogue itself and

We need some sort of translation (refined search) of it to, to all available data on DestinE.

destination-earth / DestinE_ESA_GFTS