Compile tentative application case checklist

uekerman commented 3 months ago

To standardize application cases, make them reproducible, and make them easier to test automatically, we compile a checklist of quality criteria similar to the adapter standardization. We again want to list all "preCICE checked" application cases on the preCICE website. The cases itself, however, do not have to be hosted under the preCICE organization. We neither want to provide DOIs ourselves. Instead, users could host their cases at any data repository (e.g., Zenodo or DaRUS) and get a DOI from there. The review process could be started by opening a pull request to the website. The checklist could contain manual and automatic checks.

Potential criteria could include:

DOI of case
contact information
with which versions of solvers, adapters, and preCICE is the case compatible
preCICE configuration passes offline check developed in work package WP1.1 and is formatted according to defined standard (automatically checked)
- could also include a fixed order of elements in the configuration
folder structure follows defined standard
run scripts following a defined standard
if results are included, exact information on how to reproduce them
Docker recipe to run case

uekerman commented 1 week ago

Outcome of the first discussion with @MakisH. Further discussion is planned for the coding days.

Main idea

We distinguish between metadata and best practices. The latter has three level, i.e. bronze, silver, and gold. We list on the website every application case that has all required metadata and at least bronze level.

Metadata

The usual things. Non-complete list:

title of the case
authors
contact information
which solvers are involved (we probably need a hierarchy here: openfoam, openfoam:com, openfoam:com:v2312)
used preCICE version
versions of dependencies that are known to work
doi (optional)
dev repo url (optional)
tick what features are used in the case from a curated list (quasi-Newton, GPU, ...)
license

Best practices

Each level has a main theme to simplify orientation. The lists are not complete yet, but should rather give some first impression.

Bronze

Findable and Accessible, i.e., the case is working and documented. Moreover, contributing the case should not require installing any additional tools (e.g. config visualizer). This is the bare minimum. We need a low entry barrier.

Solvers are available (FOSS or commercial license) or part of the case.
There is a README with:
- physics: explains the setup, boundary conditions, figure with domains
- how to run
- dependencies: at least one version for each such that everything works
preCICE configuration passes offline check (in review)
run script?
...

Silver

Interoperable, i.e., follow the standards. The case plays well with other cases from the community and feels part of the preCICE ecosystem.

configuration formatted according to defined standard (automatically checked)
- could also include a fixed order of elements in the configuration
folder structure follows defined standard, incl. precice-config.xml as name
run and clean scripts following a defined standard
.gitignore
README with standardized sections, incl. visualization of configuration
follow naming scheme
...

Gold

Reusable, Reproducible, and Integrated. We can integrate the case into our development workflows and make sure that we will not break it. Other users can easily reuse and extend the case for their own needs, including adding further solvers.

Docker recipe such that the system tests can run the case
visualization scripts or documentation
include or link results, with exact dependency versions
ASTE and system tests compatible exports
if possible, include a coarse variant (runnable on a laptop)

Other checklists

JOSS Reviews: https://joss.readthedocs.io/en/latest/review_checklist.html
FAIR software: https://fairsoftwarechecklist.net/v0.2/

BenjaminRodenberg commented 1 week ago

For my taste the topic reproducibility and reference results comes a bit late, but I also understand the argument of accessibility and a low entry barrier. However, at the moment it is theoretically possible to submit a case that just produces colorful pictures and reach the silver stage.

Another point: Does this list only refer to new application cases e.g. perpendicular-flap or do we also consider extensions like perpendicular-flap/solid-fenicsx? I'm thinking about the situation that an expert using FEniCSx wants to contribute a FEniCSx version of the case to the existing tutorial. We still would like to keep the entry barrier low but we also would like to have consistent results with the other flavors of the case. I imagine that in this situation the criteria mentioned under bronze and silver are really low hanging fruits for us while the gold level criteria are the main points where we would rely on the external knowledge.

MakisH commented 1 week ago

However, at the moment it is theoretically possible to submit a case that just produces colorful pictures and reach the silver stage.

These are also many of our tutorials.

Another point: Does this list only refer to new application cases e.g. perpendicular-flap or do we also consider extensions like perpendicular-flap/solid-fenicsx?

We need to distinguish here between tutorials and generally application cases. Our tutorials are application cases, which are also integrated into our development workflows and have some additional restrictions (e.g., showcasing some feature, not taking too long to run, being rather simplistic, etc).

If one wants to contribute a tutorial, they should still follow the tutorial contributing guidelines. After we have these checklists published, the guidelines will refer to these checklists (requiring gold level) and have a few additional requirements.

Good point about contributing a subcase. If it is really one extension of our tutorials, it should be integrated into the tutorials, with the same procedure as now. If we cannot invest the resources to make it gold, then they can still be published as an application case hosted elsewhere, or contributed to the non-reviewed community projects forum section.

uekerman commented 3 days ago

Beyond the tutorials, we should consider application cases that are published along papers. Example:

https://doi.org/10.18419/darus-1778

MakisH commented 1 day ago

I am wondering if we also need a level below bronze (with significantly reduced visibility): Something where everyone can link to any already published case, just by providing the metadata. This could allow to also list on our collection some older cases that nobody can invest the effort in updating to fit our requirements.

Such a level could be called "Gray", which would essentially mean "not (yet) reviewed".

At the same time, we need additional ways to incentivize going to higher levels. Maybe we integrate Gold cases into some test suites, we only mention new Silver and Gold cases in the workshops, or even we have some kind of community (financial) award for best new Gold-level case in the workshops.

precice / preeco-orga