search

OneArgo / ArgoVocabs

A repository for the management of issues related to vocabularies managed by the Argo Data Management Team
8 stars 0 forks source link

DEPLOYMENT_PLATFORM : unambiguous ship name and codes #2

Open matdon17 opened 4 years ago

matdon17 commented 4 years ago

The Argo metadata field DEPLOYMENT_PLATFORM is currently populated in a wide variety of ways, with different entries for the same ship even at the same DAC. The C17 ICES platform code vocabulary is already used within the Ocean OPS system, and if adopted for use inside the Argo NetCDF files, we will need to consider the form it takes.

An examples would be to use the full identifier, e.g. “SDN:C17::74JC James Clark Ross” to avoid ambiguity and support other vocabs e.g. for air-launched floats from planes

gabrielapilo commented 3 years ago

Hi Matt, from what I remember you had a link on your presentation at the ADMT2020 (day 1) for the standardised vessel names (maybe from JCOMMOPS?) Could you please share that link here?

matdon17 commented 3 years ago

BODC host the C17 vocabulary which is essentially a copy of the ICES platform codes: https://www.bodc.ac.uk/resources/vocabularies/vocabulary_search/C17/

ICES also make their version of the codes available via: https://vocab.ices.dk/

There is a relationship between BODC, ICES and Ocean OPS, the details of which I would need to dig out.

tcarval commented 2 years ago

The Argo metadata field DEPLOYMENT_PLATFORM is a human readable variable. We may enhance it with the feature URI (Universal Ressource Identifier) . This is human readable; this is machine readable : a computer code will use a simple regexp instruction to parse *uri:<>** elements.

Example:

randerson57261 commented 2 years ago

I plan on transitioning our database to the C17 vocabulary, using the "Preferred Label" field. But the "Identifier" will also be linked and available for use.

I wasn't aware that C17 existed, so we created our own vocabulary from a variety of sources.

vpaba commented 2 years ago

Hello,

After discussion with Victor and Magali at @nvs-vocabs/oceanops, we agree that Argo may benefit from implementing C17 in populating the DEPLOYMENT_PLATFORM field. The way that @tcarval proposed sounds like a good one, for the reasons he described. It would be good to know what the @nvs-vocabs/avtt thinks about the implementation of C17?

There is one issue that Victor identified, and that would be good to discuss: the NVS C17 vocabulary is syncronised with ICES, and as such, currently any new addition needs to go through the ICES/NOAA approval process. This may take several weeks. The outcome we want to avoid is to have the GDAC file checker refuse the registration of a new float (by rejecting the metadata file) because the C17 code for the new platform from which the float was deployed from (e.g. a ship of opportunity) is still pending approval.

How could we get around this issue?

vturpin commented 2 years ago

Also at OceanOPS we already use the ICES code witch is similar to C17 so should be "easy" to benefit from this evolution.

To complement the discussion, we also have a internal procedure to request ICES code any time a new ship is registered in our system. That may serve for the problem identified above.

ClaudiaAOML commented 2 years ago

Hello Violetta & all,

To avoid issues with GDAC submissions, an issue with the content in the DEPLOYMENT_PLATFORM field should result in a warning rather than a rejection.

I also think it would be good to review the rejection list for meta.nc files. The reason for this is that it sometimes takes quite a while to update a table on NVS and have that update propagate to the format checker. This results in consistent rejection of updated meta files. In such cases we submit a meta.nc file with suppressed information so that the GDAC accepts it, but the regular updates with new configuration information are rejected. That means users do not have access to such time variant information because one other field with pending updates causes a rejection.

Bye,

Claudia

On Tue, Feb 15, 2022 at 6:52 AM Violetta Paba @.***> wrote:

Hello,

After discussion with Victor and Magali at @nvs-vocabs/oceanops https://github.com/orgs/nvs-vocabs/teams/oceanops, we agree that Argo may benefit from implementing C17 in populating the DEPLOYMENT_PLATFORM field. The way that @tcarval https://github.com/tcarval proposed sounds like a good one, for the reasons he described. It would be good to know what the @nvs-vocabs/avtt https://github.com/orgs/nvs-vocabs/teams/avtt thinks about the implementation of C17?

There is one issue that Victor identified, and that would be good to discuss: the NVS C17 vocabulary is syncronised with ICES, and as such, currently any new addition needs to go through the ICES/NOAA approval process. This may take several weeks. The outcome we want to avoid is to have the GDAC file checker refuse the registration of a new float (by rejecting the metadata file) because the C17 code for the new platform from which the float was deployed from (e.g. a ship of opportunity) is still pending approval.

How could we get around this issue?

— Reply to this email directly, view it on GitHub https://github.com/nvs-vocabs/ArgoVocabs/issues/2#issuecomment-1040183324, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW5R4ZKA23RI2BJ4674NQ3TU3I5A7ANCNFSM4UAWYKWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are on a team that was mentioned.Message ID: @.***>

-- 

   Dr. Claudia Schmid
    NOAA/AOML/PHOD                  Phone: 305-361-4313
   4301 Rickenbacker Causeway      Fax:   305-361-4412
   Miami, FL 33149                 email: ***@***.***

As per the 2005 NOAA security awareness recommendations, the contents of this message are mine personally and do not necessarily reflect any position of the U. S. Government or of the National Oceanic and Atmospheric Administration (NOAA).

vturpin commented 2 years ago

Rejection should lead to improving and not degrading the meta.file.nc. But indeed, if the file checker or NVS vocab is not up to date rejection become conterproductive.

Maybe the warning Vs rejection should be align with the approval stage of a new collection ? related to #33 ?

vturpin commented 2 years ago

It seems we are reaching some kind of agreement here :

"The Argo metadata field DEPLOYMENT_PLATFORM can be enhanced with the feature URI. This is human readable and machine readable. Example: DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.nerc.ac.uk/collection/C17/current/33RO/" No uri will lead to a warning in the format checker."

BUT, because is not possible to have "rejection" in the case of new "DEPLOYMENT_PLATFORM", warnings may lead to no impact on the metadata files... Is there a way to encourage users to fill the uri when it exist in C17 and allow "DEPLOYMENT_PLATFORM" with no C17 collection yet ?

tcarval commented 2 years ago

I my opinion, there is no strict way to oblige users to fill deployment_platform with C17 codes. What we should do is regular monitoring (twice a year, before admt and ast), list the "good" and the "could be better" DACs

vturpin commented 2 years ago

I think this is a monitoring we could do at OceanOPS, modulo some evolution in the index files we are using for such statistics. We could also set up a metadata QC feedback on ships based on this analysis...

randerson57261 commented 2 years ago

If C17 is a unique list, why are we adding the URI? We don't add the URI in any other field.

If it's a unique list, a user should be able to look up the entry in C17 to get the other associated information.

vpaba commented 2 years ago

Hi @randerson57261, I think the reason we would need both C17 concept ID (URI) alongside the 'preferred label' is that the preferred label is not unique - only the URI is.

This is because a platform may maintain the same name after being decomissioned for e.g. undergoing refurbishment, in which case a new URI would be assigned to it but the 'preferred label' would be the same. Moreover, there may be indentical 'preferred labels' for different platforms.

See for example the ship with preferred label 'DISCOVERY' is associated with four different URIs: http://vocab.nerc.ac.uk/search_nvs/C17/?searchstr=discovery&options=identifier,preflabel,altlabel,definition,status_accepted&rbaddfilter=inc&searchstr2=

randerson57261 commented 2 years ago

That makes sense, thank you for the clarification.

Could we use the "Identifier" field (ICES code correct?) instead of the URI? Something like "Discovery, 74E3"

An advantage here is that if we have the ICES code for a deployment platform, but it is not yet in C17, it would still be valid metadata even if the file checker throws a warning. In the case that we use the URI, if a platform is not in C17, we will have to revise the metadata at some point once it is added (less likely to actually happen).

One small hesitation with the URI is that it may change (it shouldn't given it's a URI! but who knows in 10 years).

tcarval commented 2 years ago

If C17 is not (yet) available, you may provide the ICES URI ? With the uri, you allow machine to machine processes, and for humans, someone who never heard about ICES platform codes or C17 have a full description using the URI . A URI should permanently identify a resource, even if the resource is moved or deleted.

anthoninlize commented 2 years ago

Simple question: how often C17 sync with SHIPC from ICES? Following @tcarval 's comment, the advantage of using an URI as well is to be able to either fill in the C17 URI (which mirrors SHIPC from ICES) or the SHIPC URI from ICES... Since the URI specifies the namespace, that is not an issue. And why not then use the SHIPC URI directly ? SHIPC http://vocab.ices.dk/?ref=315 74E3 http://vocab.ices.dk/?CodeID=127475

vturpin commented 1 year ago

Let's reactivate this discussion a bit.

Here we are : Use “prefered label + uri form C17 or ICES SHIPC" for DEPLOYMENT_PLATFORM Following the scheme PLATFORM_DEPLOYMENT = " uri:"

Exemple: DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.nerc.ac.uk/collection/C17/current/33RO/" DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.ices.dk/?CodeID=127475"

Regular monitoring by OceanOPS by DACs; "good" : more than 90% of deployment_platform with valid C17 uri or ICES SHIPC uri "could be better" : less than 90% "why ?" : 0%

Following the procedure suggested by @vpaba here https://github.com/nvs-vocabs/ArgoVocabs/issues/33. We are between step 3: "a draft a proposal for discussion and agreement" and 4: "submit proposal to ADMT for official endorsement" Shall we submit this proposal to the ADMT on Wednesday?

BrianKingNOC commented 1 year ago

@tcarval @vturpin

I came across this when looking for something else

Here is my view in case it comes up at ADMT

I would be OK with DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.nerc.ac.uk/collection/C17/current/33RO/"

But I am less keen on DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.ices.dk/?CodeID=127475"

They both have the advantage of including the current ship name "Ronald H. Brown", (which is sometimes not unique), and then a unique uri. Everything to the left of the uri is for a person to read. Everything to the right of the uri is for a machine to read.

The NVS code uses the unique 4-character codes, which are somewhat human readable. I know that in 33RO, the 33 refers to USA, and with some familiarity I may also know that RO is the Ron Brown. So with some familiarity errors at a DAC in selecting the uri for the ship name should be minimal.

But can anyone see at a glance what is wrong with the second form using the ices code ? Not obvious straight away ? The answer is that the ices uri link is to the wrong ship.

I know it was only an example, but the point is an important one: errors are very hard to spot. In fact, 127475 refers to the UK Discovery used from 1962 to 2012.

vturpin commented 1 year ago

I don't have strong opinion about wich collection we should use. I am fine with both or only a single one.

But this exemplar brings us back to the control of the content of a metadata. In that space, OceanOPS is manually controling the deployment platforms for any float deployment. This will continue after the implementation of this issue. Any approach (ICES, C17 or both) will anyway reduce the number of errors in the metadata declaration and facilitate the control of this metadata. Allowing maybe a feebback to DAC when we identify an potential error. Such feedback is not really feasible at the moment since there is no control rule about "platform_deployment".

tcarval commented 1 year ago

The proposal is to populate the deployment_platform variable with human readable and machine readable unambiguous URI.

DEPLOYMENT_PLATFORM=“\<Human readable ship name> \<ship universal ressource identifir - uri>"

DEPLOYMENT_PLATFORM=“Ronald H. Brown uri:http://vocab.nerc.ac.uk/collection/C17/current/33RO/"

delphinedobler commented 1 year ago

Hello @tcarval @apswong,

the format should also be adapted accordingly. Currently, the character length of the field is STRING32, which is not enough if we add the URI. For instance "Ronald H. Brown uri:http://vocab.nerc.ac.uk/collection/C17/current/33RO/" represents a length of 72 characters. From NVS C17 table, the max length of a ship name is 40 characters. If we add the 56 characters from the URI + the space in-between, the maximum length will be 97 characters. I reckon using STRING128 would be fit.

We also need to deal with ships that are not under C17: how do we choose to manage them ? For instance, on Coriolis, we have ships of opportunity (sail boats, fishing vessels) or military boats general naming, which are not in C17. I reckon they might not be included in this table. For coriolis, this represents approximately 200 floats/100 deployment platform names.

vturpin commented 4 months ago

Following discussions with NVS, we have noted the proposal to use the format DEPLOYMENT_PLATFORM=“ <ship universal resource identifier - URI>”.

As mentioned by @delphinedobler, we need to consider ships deploying floats without any ICES code, IMO number, or Call Sign.

OceanOPS can provide NVS with a list of ships via API to create a new controlled vocabulary for Argo. It has been agreed that using a schema (a subset of an existing collection) is preferable to creating a new collection.

In this context, we propose the following solution:

1) OceanOPS will provide, through an API, a list of ships with ICES codes that have already deployed Argo floats. This list will be served by NVS similarly to other R lists to Argo users and will be a subset of C17. Limiting this list aims to facilitate its use.

2) The Argo user manual should redefine "DEPLOYMENT_PLATFORM" as follows: char DEPLOYMENT_PLATFORM(STRING128); DEPLOYMENT_PLATFORM:long_name = "<Human readable ship name> <ship universal resource identifier - URI>"; DEPLOYMENT_PLATFORM:_FillValue = " "; Comment: ship name and URI, from collection RXX Example: L’ATALANTE uri:https://vocab.nerc.ac.uk/collection/C17/current/35A3/

3) The checker should verify that:

In both cases, the checker should warn the user if the PLATFORM_NAME is not correctly filled and encourage the user to check the new RXX list for the deployment platform.

4) For any new ships, ships without ICES codes, or mismatches between URI and ship name, OceanOPS will continue harmonizing ship names and request ICES codes when necessary to complete the list of ships.

gwemon commented 4 months ago

I have just been through the comments in this ticket following a meeting with Vi and Dani, and thought I would bring some clarification about the history of C17 and the ICES SHIPC vocabularies and the current status of the workflows.

gwemon commented 4 months ago

FYI: The list of deprecated C17 codes can be found here: http://vocab.nerc.ac.uk/search_nvs/C17/?searchstr=&options=identifier,preflabel,altlabel,status_deprecated&rbaddfilter=inc&searchstr2= And it seems that many have now also been deprecated in SHIPC (https://vocab.ices.dk/?ref=315) so maybe the difference between the two vocabs is no longer as large as it used to be but until ICES and BODC find the time and resource to review the legacy there will remain some uncertainty.

gwemon commented 4 months ago

Sorry I closed that ticket by mistake! @vturpin regarding your proposal to create a new collection for the platforms that have deployed Argo floats, could you provide us with the use cases that would necessitate creating and maintaining a specific subset of C17 for Argo? Is it to facilitate Discovery when people are creating the metadata ? or to facilitate validation? something else? this could be quite costly and there might also be different ways of addressing the use cases that would require less maintenance.

danibodc commented 1 month ago

Decision to create new DEPLOYMENT_PLATFORM collection limited to all Argo platforms, and map to existing C17 entries. Map to new C17 entries as and when they are approved by ICES.

vturpin commented 1 month ago

ocean-ops.org/api/1/data/ship/?include=["id","name","country.name","country.code2","country.code3","imo","mmsi","callSign","ref"]&exp=["ptfDeployments.ptfs.networkPtfs.network.name=%27Argo%27"]

@danibodc Is this enough or do you prefer csv file ? Magali is working on it anyway.

tcarval commented 1 month ago

In my op

ocean-ops.org/api/1/data/ship/?include=["id","name","country.name","country.code2","country.code3","imo","mmsi","callSign","ref"]&exp=["ptfDeployments.ptfs.networkPtfs.network.name=%27Argo%27"]

@danibodc Is this enough or do you prefer csv file ? Magali is working on it anyway.

In my opinion, Dani creates the table, and you Victor or Magali fill the table.

vturpin commented 1 month ago

I've send a csv file by email in the meantime. If you need more or less information, we will redo the extract.

delphinedobler commented 1 month ago
vturpin commented 1 month ago

Regarding the two first items I like the name "R29 - Argo deployment ship" if R29 is still available. The collection has been send to Dani.