softwaresaved / fuji

FAIRsFAIR Research Data Object Assessment Service
MIT License
0 stars 1 forks source link

[Feature]: [FRSM-14] Does the software come with test cases to demonstrate it is working? #14

Open karacolada opened 7 months ago

karacolada commented 7 months ago

D5.2 p20+p30

Detailed Description

The provision of test cases improves confidence in the software.

Domain-agnostic comments

Most programming languages have commonly associated test frameworks. The specific definition of what constitutes adequate testing is often defined by community norms. It is hard to check the relevance / correctness of this information, but it is possible to automatically check for existence.

CESSDA comments

See Software Maturity Levels (SML) for: CMA9 - Verification and Testing and CMA7 Standards Compliance. CESSDA periodically runs the SQAaaS tool against its publicly accessible repositories and displays the results via a badge in the README file.

Context

R1: Software is described with a plurality of accurate and relevant attributes.

Possible Implementation

domain-agnostic

requirements Software source code
method Check for the presence of automated tests
essential Tests and data are provided to check that the software is operating as expected
important Automated unit and system tests are provided
useful Code coverage / test coverage is reported

CESSDA

requirements Software source code
method Check the README file.
essential The README file includes badges that link to a comprehensive code quality assessment tool (SonarQube) and automated build tool (Jenkins).
important CMA9-SML5 - Demonstrable usability: A production system has been tested and validated through successful use of the application.
CMA7-SML5 - Demonstrable usability: Compliance with open or internationally recognised standards for the software and software development process, is evident and documented, and verified through testing of all components. Ideally independent verification is documented through regular testing and certification from an independent group.
useful The README file badges indicate the status of the tests and other code quality metrics. The repository contains a subdirectory containing code for the test cases that are run automatically.
karacolada commented 1 month ago

Skeleton evaluator: https://github.com/softwaresaved/fuji/blob/1778e730c80852f691e135a1b288bc2796914c46/fuji_server/evaluators/fair_evaluator_test_cases.py#L10