search

rd-alliance / FAIR-data-maturity-model-WG

https://www.rd-alliance.org/group/fair-data-maturity-model-wg/case-statement/fair-data-maturity-model-wg-case-statement
13 stars 3 forks source link

Indicators prioritisation for Findability #30

Closed bahimc closed 4 years ago

bahimc commented 5 years ago

Dear members of the Fair data maturity model Working Group,

As a next step towards a common set of core assessment criteria, we started to explore the prioritisation of the indicators derived from the contributions to this very repository. As a result, all the indicators have been ranked according to three degrees of priority;

In the following Google spreadsheet, you can see the i) indicators in their final version and ii) their priorities.

F

Please have a look at the spreadsheet and let us know what are your thoughts, suggestions and concerns with regards to the ranking we attempted to make.

Thanks in advance for your contribution!

makxdekkers commented 5 years ago

@helenp A new proposal is for the indicators for F4 to be collapsed into one:

Metadata is offered/published/exposed in such a way that it can be harvested and indexed.

This removes the identification of where the metadata is indexed. Would that answer your concern?

helenp commented 5 years ago

Yes, that works. This would also account for the case where some data was exposed outside a repo, Thanks.

CaroleGoble commented 5 years ago

forgive if I missed something, but...where are the concrete examples for each indicator (that illustrate fail and succeed) and is there a synthesised summary of the arguments for the decision? It seems embedded in these discussions

JolandaS commented 4 years ago

forgive if I missed something, but...where are the concrete examples for each indicator (that illustrate fail and succeed) and is there a synthesised summary of the arguments for the decision? It seems embedded in these discussions

Examples would be very welcome, as the indicators are not always clear on themselves.

JolandaS commented 4 years ago

Recently we scored these indicators (v0.01) on a transcriptomics dataset for the FAIRplus project. For the most part it went really well. We did have some questions on the exact meaning of certain indicators:

A1.1-01D: with free is it meant "open" or as in payment? I3-02M: what is "sufficiently qualified"? If it leads you back to other metadata it is sufficient? R1-01M: this highly depends on what you want to reuse it for? In our case, reuse to try new mapping algorithm: yes (format is standardized); reuse in your own bigger experimental setup: probably no (as on SRA no information on cell media, experimental protocol,...) R1.3-01M: community standards seems to be overlapping with R1-01M. What is the difference with domain/discipline standards?

Some of these questions would already be answered by adding and example I think, as indicated before. Looking forward to any answers and hearing what you think.

makxdekkers commented 4 years ago

@JolandaS Some of your questions are about the wording of the principle. On the GO-FAIR website there are further explanations of the terms used: A1.1: "free (no-cost)" I3: "A qualified reference is a cross-reference that explains its intent"

There is indeed overlap between R1 and R1.3. The mention of 'sufficient metadata' in the indicator for R1 is to satisfy the requirement of "plurality and relevance of attributes" in the principle, while the indicators for R1.3 focus on the "standard" aspect. The word 'standard' does not appear in the definition for R1, but was added later based on comments in the meeting on 18 June 2019 (see also https://github.com/RDA-FAIR/FAIR-data-maturity-model-WG/issues/26#issuecomment-504982246) because the richness of metadata is dependent on the community. Would it be clearer if the text of the indicators were aligned to mention 'community standard' instead of 'domain/discipline-specific metadata standard' in R1?

andrasholl commented 4 years ago

R1.3-01M: community standards seems to be overlapping with R1-01M. What is the difference with domain/discipline standards?

Using the same language (community standard) would be helpful indeed.

Andras Holl

----- Original Message ----- From: "makxdekkers" notifications@github.com To: "RDA-FAIR" FAIR-data-maturity-model-WG@noreply.github.com Cc: "holl andras" holl.andras@konyvtar.mta.hu, "Mention" mention@noreply.github.com Sent: Monday, 30 September, 2019 21:08:48 Subject: Re: [RDA-FAIR/FAIR-data-maturity-model-WG] Indicators prioritisation for Findability (#30)

R1.3-01M: community standards seems to be overlapping with R1-01M. What is the difference with domain/discipline standards?

@JolandaS Some of your questions are about the wording of the principle. On the GO-FAIR website there are further explanations of the terms used: A1.1: "free (no-cost)" I3: "A qualified reference is a cross-reference that explains its intent"

There is indeed overlap between R1 and R1.3. The mention of 'sufficient metadata' in the indicator for R1 is to satisfy the requirement of "plurality and relevance of attributes" in the principle, while the indicators for R1.3 focus on the "standard" aspect. The word 'standard' does not appear in the definition for R1, but was added later based on comments in the meeting on 18 June 2019 (see also #26 (comment)) because the richness of metadata is dependent on the community. Would it be clearer if the text of the indicators were aligned to mention 'community standard' instead of 'domain/discipline-specific metadata standard' in R1?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/RDA-FAIR/FAIR-data-maturity-model-WG/issues/30#issuecomment-536707162 -- Holl András informatikai főigazgató-helyettes / deputy director (IT) MTA Könyvtár és Információs Központ / MTA Library and Information Centre

JolandaS commented 4 years ago

I agree, "community standard" for both would be better.

kitchenprinzessin3880 commented 4 years ago

@makxdekkers There are three versions in the google spreadsheet. Can you please confirm the stable version? is version work in progress?

bahimc commented 4 years ago

@kitchenprinzessin3880 thanks for your message. The stable version is v0.03, which is accessible via this link.

kitchenprinzessin3880 commented 4 years ago

@RDA-FAIR v3 - noted, thanks. The reason for the change of the priority levels (from v2.0 to v3.0) is unclear. How do you determine the indicators priority levels in V3.0?

bahimc commented 4 years ago

@kitchenprinzessin3880 as presented during Workshop 5, we have issued a survey about the prioritisation about the indicators. V0.03 incorporates the results of the survey as for the prioritisation of indicators.

kitchenprinzessin3880 commented 4 years ago

@kitchenprinzessin3880 as presented during Workshop 5, we have issued a survey about the prioritisation about the indicators. V0.03 incorporates the results of the survey as for the prioritisation of indicators.

ok noted. thanks!

bahimc commented 4 years ago

Dear contributors,

As reminded during our fifth workshop in Helsinki, the current state of the indicators - and their priorities, as of early October 2019, is now frozen. The current indicators will be used in a testing phase where owners of evaluation approaches are invited to compare their approaches (questionnaires, tools) against the indicators. More information concerning the testing phase will be communicated during workshop #6, on the 4th of December.

As such, the current set of indicators can be seen as an ‘alpha version’. In the first half of 2020, the indicators may be revised and improved, based on the results of the testing.

Should you have any question or feedback, please reply to this comment.