ICES Data Centre Data Validation review

[neil-ices-dk]

We are internally looking at all the data validation approaches that we currently use across all the data types; over time we have used SQL, bespoke apps, XML, Schematron etc. The idea being to see if we should be taking a new consolidated approach, and finding out if so, what that approach might look like from a process/technology perspective.

As part of the analysis steps, we are looking at current and best practices from the marine data management community. We initially started with an extensive survey, but after feedback from Jens, Dave and Sjur we have slimmed it down to a more basic canvassing, with the idea that we could then follow-up with specific respondents if their answers have information that we would like to expand on. To that end, the draft survey is here Data Validation Survey

the timeline below is a bit behind schedule, but it is indicative of the steps

[mehdiabbasi]

The result of the Data validation survey and also new New Data Validation Framework Proposal​ will be presented at the May DIG meeting https://community.ices.dk/Committees/DIG/DIG%202023/05.%20Presentations/ICES%20new%20data%20validation%20framework%20proposal.pptx

[davidcurrie2001]

@mehdiabbasi are there any updates on this topic? Are you planning to present something at the DIG meeting in May?

[mehdiabbasi]

@davidcurrie2001 yes, I am planning to update DIG about Data Validation Phase II progress but this will be included in the half a day DC presentation.

[mehdiabbasi]

@davidcurrie2001 We will present the progress on Data Validation framework design - Phase II in this presentation DIG-Data validation framework-PhaseII. We will also present data Validation Requirements documents and would like to ask DIG to review these requirements and give us feedback on each feature, and whether there are any requirements needed by the community that are missing

[Ingeborg1312]

General comment on the report presented at DIG 2024: it may help to define 'target audience' to all paragraphs in chapter 2, to get clear who could provide feedback, or be asked as test panel/reviewers, especially when part of the target audience is outside the ICES Data Centre (for example DIG, governance groups, data submitters)

[Ingeborg1312]

General comment on the report: for improved overview is may be worth to organise the different bullets in a topical manner per paragraph. Eg. in 2.1.4 the first, second and sixth bullet are about validation reports, and the other about formats.