Data Template Update (version 2)

[neil-ices-dk]

following on from #49 and #239

[neil-ices-dk]

vocab changes implemented and linked to format; @cmspinto to send latest version of template to Joana; and also inform Nikki when ready to update data standard #245 ;

[NikkiTaylorJNCC]

@cmspinto @pcrjoana I am just updating the JCDP data standard in line with the latest version of the data format template - The data right holds contact field still seems to exist in the template? We discussed removing that contact, and just having the contact for the data custodian, mainly to remove the need for data owners to be in EDMO as this may be off putting and reduce data submissions. The data custodian contact description also still referenced the previous field label - 'DataOriginator' so could that please be amended? Second point - under 'Cue' we have 'breach' but under 'behaviour' we have 'breaching clear out of the water' which isn't supported by the description in the data standard - can the behaviour vocab be the same as cue, and just have 'breach'?

[NikiClear]

If the template is ready, could the latest version be made available to the portal webpages before the workshops please? (current version available is v1.10).

[neil-ices-dk]

@pcrjoana is the v2.0 ready for upload?

[pcrjoana]

@neil-ices-dk Just sent it over! We will have to sort out the labels in the Behaviour vocab, as some of the codes have a "cetaceans" prefix, which is coming from the ESAS project. This issue will be raised with the vocabulary management group.

[neil-ices-dk]

new version (2.0) is published on website https://www.ices.dk/data/Documents/Cetaceans/Cetaceans_data_template.zip

[NikkiTaylorJNCC]

@neil-ices-dk Just sent it over! We will have to sort out the labels in the Behaviour vocab, as some of the codes have a "cetaceans" prefix, which is coming from the ESAS project. This issue will be raised with the vocabulary management group.

@pcrjoana Joana - as long as we can keep the simplified header in the data standard and the data template i'm not too worried - we have already adjusted the breaching category to not show 'clear of the water' as it is in the vocab.

[NikkiTaylorJNCC]

@neil-ices-dk @pcrjoana @cmspinto @NikiClear so are we essentially ready for the system to go live to accept data?? Our next meeting isn't until the 28th, I am on leave next week - it would be great if we could open the system up this week and flag it's ready. Need to make sure:

• the template on the landing page is ok (@pcrjoana - do we need to wait for the vocab meeting outcome?)
• the system is clear of any test data @cmspinto
• JCDP data standard is made available (we can circulate via email in advance of it getting published online) @NikiClear
• Anything else?

[pcrjoana]

@neil-ices-dk @pcrjoana @cmspinto @NikiClear so are we essentially ready for the system to go live to accept data?? Our next meeting isn't until the 28th, I am on leave next week - it would be great if we could open the system up this week and flag it's ready. Need to make sure:

• the template on the landing page is ok (@pcrjoana - do we need to wait for the vocab meeting outcome?)
• the system is clear of any test data @cmspinto
• JCDP data standard is made available (we can circulate via email in advance of it getting published online) @NikiClear
• Anything else?

There is need to wait for the vocab meeting but we can't open submissions yet. I'll work on the basic data validation tomorrow, which Carlos has already started. Do you have any checks that need to be added that aren't related to the format? These would be related to species distribution or numbers, basically things that aren't biologically possible. Carlos will know better, but I think we can start accepting data after the data validation is in place.

[NikiClear]

Hi @neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto

I’m afraid I have some bad news. We’ve had a message from Lucy with some questions about the format and it looks like there may be some points mentioned during the workshop which might have fallen through the net. I’m sorry these weren’t picked up on before, I hope they won’t cause too much of a headache!!

Q: Unable to record multiple precipitations e.g. rain and fog - Joana said this is possible and it will be added, is this still the case? From my understanding, it isn’t currently possible to record multiple precipitation types. Would it be possible to allow multiple options to be entered?

Q: Will Sleet be added as a precipitation option? Would it be possible to add ‘ Sleet’ as an additional option for the precipitation vocabulary, please?

Q: Radial distance - does this now accept decimal places?

@pcrjoana – is there any update on progress of getting the new ORCA shipcodes approved? Have you had any others come forward to get their shipcodes added?

[pcrjoana]

Hi @neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto

I’m afraid I have some bad news. We’ve had a message from Lucy with some questions about the format and it looks like there may be some points mentioned during the workshop which might have fallen through the net. I’m sorry these weren’t picked up on before, I hope they won’t cause too much of a headache!!

Q: Unable to record multiple precipitations e.g. rain and fog - Joana said this is possible and it will be added, is this still the case? From my understanding, it isn’t currently possible to record multiple precipitation types. Would it be possible to allow multiple options to be entered?

Q: Will Sleet be added as a precipitation option? Would it be possible to add ‘ Sleet’ as an additional option for the precipitation vocabulary, please?

Q: Radial distance - does this now accept decimal places?

@pcrjoana – is there any update on progress of getting the new ORCA shipcodes approved? Have you had any others come forward to get their shipcodes added?

Hello! Q1: It is possible, but they need to register multiple efforts. They can have multiple effort records in one survey, but they can only link one sighting to one effort. Does this solve it, or do they need codes for multiple options (eg rain and fog)? If so, please forward a list of the codes needed.

Q2: Yes, it can be. Can you compile a list of all codes needed to be added to the precipitation vocabulary (see Q1)?

Q3: It will in a few minutes. ;-)

Have ORCA entered all the ships? They were going to send me a list with their ship names once they added, but they haven't done this yet. So far only Laia has submitted full ship information, that I know of.

[NikiClear]

Thanks Joana! Q1: I don't think registering multiple efforts would be a solution, because of the sighting link issue as you said. I'll ask Lucy for the combinations needed for precipitation they would need and I will send you a list.

Q3: I'm delighted this can be done so quickly, I was worried it might have been a big issue.

I'll get back to ORCA and ask about their list of ship name - hopefully they will be back in touch with you soon.

[NikiClear]

Q2: Yes, it can be. Can you compile a list of all codes needed to be added to the precipitation vocabulary (see Q1)?

Hi @pcrjoana. Here is a list for the precipitation codes as requested. I'll also update the data standard accordingly.

Code | Description -- | -- NR | Not recorded N | None F | Fog H | Hail R | Rain S | Snow St | Sleet RF | Rain and Fog HF | Hail and Fog SF | Snow and Fog StF | Sleet and Fog

[NikiClear]

@neil-ices-dk @pcrjoana @cmspinto @NikiClear so are we essentially ready for the system to go live to accept data?? Our next meeting isn't until the 28th, I am on leave next week - it would be great if we could open the system up this week and flag it's ready. Need to make sure:

• the template on the landing page is ok (@pcrjoana - do we need to wait for the vocab meeting outcome?)
• the system is clear of any test data @cmspinto
• JCDP data standard is made available (we can circulate via email in advance of it getting published online) @NikiClear
• Anything else?

There is need to wait for the vocab meeting but we can't open submissions yet. I'll work on the basic data validation tomorrow, which Carlos has already started. Do you have any checks that need to be added that aren't related to the format? These would be related to species distribution or numbers, basically things that aren't biologically possible. Carlos will know better, but I think we can start accepting data after the data validation is in place.

@neil-ices-dk @pcrjoana @cmspinto @NikiClear @NikkiTaylorJNCC Distribution and numbers may be tricky as there is always a possibility of oddities being seen. But could we have a check to ensure no coordinates are on land as a basic check at least.

Is validation just a pass / fail? Or is there options to flag records which may need checking? i.e. if 1000 fin whales were recorded as a sighting. This would be highly unlikely but not technically impossible.

[cmspinto]

Hi Nikki,

both checks are possible.

a) It is possible to check that the coordinates are not on land

b) It is also possible to have a warning if a value seems unlikely. Like if species = "wales" and NoIndividuals > 100

@pcrjoana has been working on the checks and you can see them here: http://datsu.ices.dk/web/rptChk.aspx?Dataset=149

[NikiClear]

Hi Nikki,

both checks are possible.

a) It is possible to check that the coordinates are not on land

b) It is also possible to have a warning if a value seems unlikely. Like if species = "wales" and NoIndividuals > 100

@pcrjoana has been working on the checks and you can see them here: http://datsu.ices.dk/web/rptChk.aspx?Dataset=149

Thank you for this and for the link to the checks. Lets include the coordinates no on land check. Nikki and myself will think about other appropriate biological validation rules when Nikki is back. It's very useful to have a warning for these.

[neil-ices-dk]

MaxGroupSize and MinGroupSize; optional but conditional; if they are not filled, a warning is to be made saying this is missing.

[neil-ices-dk]

Q2: Yes, it can be. Can you compile a list of all codes needed to be added to the precipitation vocabulary (see Q1)?

Hi @pcrjoana. Here is a list for the precipitation codes as requested. I'll also update the data standard accordingly. Code Description NR Not recorded N None F Fog H Hail R Rain S Snow St Sleet RF Rain and Fog HF Hail and Fog SF Snow and Fog StF Sleet and Fog

to add to new version of data format template (v 2.01)

[neil-ices-dk]

version 2.01 now published on cetacean portal https://www.ices.dk/data/Documents/Cetaceans/Cetaceans_data_template.zip

[NikkiTaylorJNCC]

@neil-ices-dk - request made to change the speed of vessel field from mandatory to optional, as not all providers record this information. Can we agree this here rather than wait for the next sprint? Discussed in the workshop 3rd March by @pcrjoana @NikiClear @cmspinto and @NikkiTaylorJNCC

[neil-ices-dk]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

[NikkiTaylorJNCC]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

Great! Yes to all of the above. Thanks.

[NikiClear]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

The data standard has been updated. Can I confirm that the updated version of the template will be v2.02?

[neil-ices-dk]

Hi Nikki, yes the version will be 2.02 as 2.01 was formally published.

[NikiClear]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

@neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto What is the timeframe to make these changes? We're hoping to notify the data providers that the database is ready in the next couple of days if possible.

This is not as urgent as the changes to the vessel speed and platform height fields to be made optional. But we add 'Lagenorhynchus' AphiaID 137020 to the species vocab please?

[pcrjoana]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

@neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto What is the timeframe to make these changes? We're hoping to notify the data providers that the database is ready in the next couple of days if possible.

This is not as urgent as the changes to the vessel speed and platform height fields to be made optional. But we add 'Lagenorhynchus' AphiaID 137020 to the species vocab please?

The code is added to the vocabulary, and the changes to the format are made. Is there a common name for Lagenorhynchus so I can add it to the template?

[NikiClear]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

@neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto What is the timeframe to make these changes? We're hoping to notify the data providers that the database is ready in the next couple of days if possible. This is not as urgent as the changes to the vessel speed and platform height fields to be made optional. But we add 'Lagenorhynchus' AphiaID 137020 to the species vocab please?

The code is added to the vocabulary, and the changes to the format are made. Is there a common name for Lagenorhynchus so I can add it to the template?

There isn't a common name for this genus. But if you need one, perhaps something like white-beaked/white-sided dolphin. @NikkiTaylorJNCC do you have any thoughts?

[pcrjoana]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

@neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto What is the timeframe to make these changes? We're hoping to notify the data providers that the database is ready in the next couple of days if possible. This is not as urgent as the changes to the vessel speed and platform height fields to be made optional. But we add 'Lagenorhynchus' AphiaID 137020 to the species vocab please?

The code is added to the vocabulary, and the changes to the format are made. Is there a common name for Lagenorhynchus so I can add it to the template?

There isn't a common name for this genus. But if you need one, perhaps something like white-beaked/white-sided dolphin. @NikkiTaylorJNCC do you have any thoughts?

If there is none, we can leave it blank. :-)

[NikkiTaylorJNCC]

sounds good, so we need an update to:

• data standard (Nikki)
• data template (Joana/Neil)
• database (Carlos)
• DATSU (Joana)

@neil-ices-dk @pcrjoana @NikkiTaylorJNCC @cmspinto What is the timeframe to make these changes? We're hoping to notify the data providers that the database is ready in the next couple of days if possible. This is not as urgent as the changes to the vessel speed and platform height fields to be made optional. But we add 'Lagenorhynchus' AphiaID 137020 to the species vocab please?

The code is added to the vocabulary, and the changes to the format are made. Is there a common name for Lagenorhynchus so I can add it to the template?

There isn't a common name for this genus. But if you need one, perhaps something like white-beaked/white-sided dolphin. @NikkiTaylorJNCC do you have any thoughts?

If there is none, we can leave it blank. :-)

Happy to leave this blank

[neil-ices-dk]

Joana has sent the completed template v2.02 which will be uploaded straight after the meeting

[neil-ices-dk]

@NikkiTaylorJNCC v2.02 live on web