I like!

[kjappelbaum]

I feel it could be similar to checkcif, i.e. highlight potential issues for an expert reviewer (and required by journals).

I think one could check off many of the items on the JOSS reviewer checklist automatically.

Additionally, one might try:

• Run the installation with the provided commands, afterward check that one can import the module
• Check if there is a test suite (that is run automatically on GH actions, Travis, ...)
• Check if one can create issues ;)

As you suggested, one might want to also add subject-specific checks. For instance, for ML

• One can check if there is some link to a dataset on a repository
• Check if there are model checkpoints

Quite domain specific would then be to check if the train/test set are indeed independent.

[kjappelbaum]

Here's a tool that might already address some of the checkmarks https://github.com/fair-software/howfairis

[cthoyt]

I saw that. It doesn’t do what I want, and I don’t like how it does it. With this kind of stuff I would much rather rebuild it from scratch.