metrics and compliance dashboard

[schristley]

We will run automated queries regularly to monitor for new studies and/or data to be processed. Source data does not always adhere to standards, and metadata quality can be poor or missing in public archives. Thus, we will implement a dashboard that automatically reports compliance metrics generated during curation and validation (Aims 1.4, 2.4), including metadata completeness and accuracy, ontology and standards usage, and data formatting. Similar to the OBO Dashboard (http://dashboard.obofoundry.org/dashboard/index.html), the dashboard will flag issues for human review, provide reports and instructions for how to address the issues, and provide guidance for community engagement by pointing to common issues.

• [ ] Aim 1.1 critical metrics: the proportion of entities and DEs accommodated in the semantic model and standards.
• [ ] Aim 1.2 Critical metric: the proportion of DEs incorporated into the CDM. The number of repositories that harmonize to the CDM. The proportion of DEs in the CDM accommodated by each repository.
• [ ] Aim 1.3 Critical metric: the proportion of processes incorporating the interoperable standards and CDM; proportion of data and knowledge that is compliant; number of compliance metrics reported in the dashboard.
• [ ] Aim 1.4 Critical metric: the number of repositories that implement the validation processes. The proportion of DEs covered by the validation processes. The proportion of data within each repository that is covered by the validation processes. The number of processes that have been automated.
• [ ] Aim 2.1 Critical metric: the number of DEs identified and categorized.
• [ ] Aim 2.2 Critical metric: proportion of DEs for which integration rules have been designed and for which integration has been automated and deployed.
• [ ] Aim 2.3 Critical metric: the number of knowledge-enriching algorithms integrated.
• [ ] Aim 2.4 Critical metric: the number of validation processes and rules. The proportion of DEs covered by validation processes/rules. The proportion of data/metadata in the knowledgebase covered by validation processes/rules. The number of processes that have been automated.
• [ ] Aim 2.5 Critical metric: the proportion of CDM DEs queryable through the API. The proportion of DEs in each repository queryable through the API. The number of use cases (Aim 3.1) answered with an API query.
• [ ] Aim 2.6 Critical metric: the number of rules developed for propagating changes. The number of repositories that implement the propagation rules. The number of DEs that require provenance and versioning. The number of repositories that implement provenance and versioning.
• [ ] Aim 3.1 Critical metric: the number of communities and experts engaged. The number of use cases collected. The number of health/disease areas covered.
• [ ] Aim 3.2 Critical metric: the number of repositories that have enhanced their existing GUI. The number of use case workflows published on the AKC website.
• [ ] Aim 3.3 Critical metric: the number of publications. The number of presentations. The number of scientific meetings attended with an exhibitor booth.
• [ ] Aim 3.4 Critical metric: the number of new training materials produced. The number of existing training materials updated to cover new functionality. The number of webinars developed and the number conducted.
• [ ] Aim 3.5 Critical metric: the quantity of feedback gathered. The number of attending and engaged users in outreach activities. The number of user groups reached and engaged.

[schristley]

Hi @jamesaoverton , I'm assigning this to you initially as we mainly want to plan out how and what the dashboard does before we get into any implementation. In the proposal, we mostly describe this as related to curation, i.e. data from public archives is dirty thus we want to flag issues. However, we should also think of metrics and how this could apply to the various processes of the component repositories and the AK.

[schristley]

Here's one rough idea. The MiAIRR standard gives a list of elements. These are also tagged in the JSON schema with different requirement levels. MiAIRR allows many fields to be left null but it would be better to have data for them. A completeness metric could automatically measure for each Repertoire and/or Study in the ADC, the number of fields that are complete (versus being left null).

That's specifically for the MiAIRR standard. The AIRR Standards encompasses more so there could be an additional completeness metric for the larger standard too.