search

OneArgo / ADMT

A repository for Argo Data Management Team activities
7 stars 1 forks source link

GDAC homogeneous index files #3

Open tcarval opened 11 months ago

tcarval commented 11 months ago

The current index files available on GDAC https://data-argo.ifremer.fr (index.txt) are not homogeneous. A richer and homogeneous series of index files should be managed.

cgourcuf commented 9 months ago

See also #16

mscanderbeg commented 9 months ago

At the ADMT-24 meeting, it was proposed that the Deep Argo index list should include all profiles from Deep floats as well as the pressure of the deepest bin to help users identify which profiles might actually be useful. Should the description be 'Deep Argo float profiles' and 'Deep Argo float trajectories'?

In general, some of the index lists may have additional information such as the deepest pressure, or eventually a flag indicating whether floats detected ice. Should these extra data be included in a similar place for the different index lists?

RomainCancouet commented 9 months ago

The current index files available on GDAC https://data-argo.ifremer.fr (index.txt) are not homogeneous. A richer and homogeneous series of index files should be managed.

  • Dedicated directory for indexes : "idx" new directory

  • Index file naming convention

    • argo-core-prof-index.csv Argo profiles, primary sampling
    • argo-core-meta-index.csv metadata files
    • argo-core-traj-index.csv trajectory files
    • argo-core-tech-index.csv technical files
    • argo-bgc-prof-index.csv BGC-Argo profiles
    • argo-bgc-traj-index.csv BGC-Argo trajectories
    • argo-deep-prof-index.csv Deep-Argo profiles
    • argo-deep-traj-index.csv Deep-Argo trajectories

hello @tcarval : it is my understanding that the suggested argo-core- files listed here would correspond to the presently used ar_indexglobal index files, i.e. contain not only core T&S (= 2500 floats of OneArgo design) but also information from other float missions (Deep and BGC). Would it be more appropriate to name them argo-one-* files?

argo-one-prof-index.csv Argo profiles, primary sampling argo-one-meta-index.csv metadata files argo-one-traj-index.csv trajectory files argo-one-tech-index.csv technical files

delphinedobler commented 5 months ago

@tcarval, @RomainCancouet @mscanderbeg @cgourcuf, I've initiated a google doc to discuss what we wish (list of fields) with the associated costs in terms of size and growing rate: https://docs.google.com/document/d/1tY2qAgZwsOXIpTltlbsJc9c8j5hL4WNg/edit I've highlighted in yellow parts that definetely deserve to be discussed/completed.

Don't hesitate to comment.

tcarval commented 5 months ago

The current index files available on GDAC https://data-argo.ifremer.fr (index.txt) are not homogeneous. A richer and homogeneous series of index files should be managed.

  • Dedicated directory for indexes : "idx" new directory

  • Index file naming convention

    • argo-core-prof-index.csv Argo profiles, primary sampling
    • argo-core-meta-index.csv metadata files
    • argo-core-traj-index.csv trajectory files
    • argo-core-tech-index.csv technical files
    • argo-bgc-prof-index.csv BGC-Argo profiles
    • argo-bgc-traj-index.csv BGC-Argo trajectories
    • argo-deep-prof-index.csv Deep-Argo profiles
    • argo-deep-traj-index.csv Deep-Argo trajectories

hello @tcarval : it is my understanding that the suggested argo-core- files listed here would correspond to the presently used ar_indexglobal index files, i.e. contain not only core T&S (= 2500 floats of OneArgo design) but also information from other float missions (Deep and BGC). Would it be more appropriate to name them argo-one-* files?

argo-one-prof-index.csv Argo profiles, primary sampling argo-one-meta-index.csv metadata files argo-one-traj-index.csv trajectory files argo-one-tech-index.csv technical files

Here is a proposal to simplify the naming convention:

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

Index name | description | comment -- | -- | -- OneArgoIndex-metadata.csv | metadata files OneArgoIndex-profileBgc.csv | profiles, bgc files OneArgoIndex-profileBgcSynthetic.csv | profiles, bgc synthetic files OneArgoIndex-profileCore.csv | profiles, primary sampling files OneArgoIndex-profileDeep.csv | deep profiles core and bgc | should we separate core and bgc ? Probably not OneArgoIndex-technical.csv | technical files OneArgoIndex-trajectory.csv | trajectory files OneArgoIndex-trajectoryDeep.csv | trajectory files deep argo | do we need this index ?

mscanderbeg commented 5 months ago

I don't see a purpose for the Deep trajectory file unless it would include a drift pressure. Eventually, we expect deep floats to park at 1000db like the rest of the Argo fleet. I think if the OneArgoIndex-profileDeep includes the parameters, we do not need to separate core and bgc floats.

delphinedobler commented 2 months ago

@tcarval, @RomainCancouet @mscanderbeg @cgourcuf, based on the messages above, comments from Megan and a discussion session I had with Thierry yesterday, I've updated the document into a version 0.2: https://docs.google.com/document/d/1pil8WHWSVje086c7hyUNk4q3RJwCqeUJ/edit?pli=1 Any comments are very welcome.

Cheers, Delphine