collection

This repository is used to manage the resources displayed on bioimage.io.

Most users will not directly dispatch the workflows defined in this reporitory, but should instead login on bioimage.io and use the front-end to interact with the bioimage.io collection.

We currently do not have a workflow for direct upload, but publicly available resource packages may be staged with a stage workflow dispatch.

Maintaining the bioimage.io Collection

In order to update or add new resources to the bioimage.io collection, they have to undergo review. Current reviewers are listed in bioimageio_collection_config.json, under section reviewers.

Reviewer Onboarding

Open a pull request, adding a person to the list of reviewers, see https://github.com/bioimage-io/collection/pull/75 for an example.
- one public email address is required
- github id can be found using https://api.github.com/users/<github_username>
Once the pull request has been merged, the new reviewer can
- accept resource drafts
- request changes on resource drafts
- upload a new version for any resource

Adding a bioimage.io resource

A bioimage.io resource is created by a user uploading a resource package. Such a resource package could e.g. be a newly uploaded model package, or notebook package. Typically, uploaders would go via bioimage.io/upload. Alternatively, any direct link to a downloadable resource package (.zip-file) would work. The latter option is reserved for members of this repository (or the bioimageio org). Once available online the resource package is staged (see Staging section), tested (see Testing section), and reviewed (see Review section).

graph TD;
    stage[stage]
    test[test]
    cr[request changes]
    publish[accept/publish]
    backup[backup to Zenodo]

    stage-->test
    test-->cr
    cr-->stage
    test-->publish
    publish-->backup

Staging

If the resource package was uploaded via the bioimage.io website, the staging of the new resource draft is initiated automatically by the uploader service. Otherwise, given a download URL to a resource package, the stage workflow needs to be dispatched manually, or via github api ("run workflow")[^1].

Staging unpacks the files from the zipped resource package to our public S3. Once unpacked, the staged resource draft is automatically tested (the test workflow is dispatched automatically at the end of the stage workflow).

Testing

Staged resource drafts are automatically tested:

Is their metadata valid?
Can test outputs be reproduced from test inputs?
Are linked URLs available?
...

Tests can also be triggered (via github api or manually) by dispatching the test workflow[^2].

Once the tests are completed, the uploader gets a notification via email that their draft is awaiting review; now a reviewer needs to take a look.

An overview of all pending resource drafts can be found at https://bioimageio-uploader.netlify.app/#/status. A draft is identified by its concept id (id from the rdf.yaml).

Review

Reviewers should check the models for technical correctness (aided by CI, see Testing section) and contents/metadata of the resource.

To this end it can be helpful to check the logs displayed at https://bioimageio-uploader.netlify.app/#/status/<concept_id>. There information about automated workflow steps and validation outcome is logged. Additionally an 'error' status may be shown if an exception occured in the GitHub workflow run producing the log.

For models, reviewers can use the model documentation as a guide.

Reviewers can:

request changes: A contributor is expected to upload an updated (fixed) draft (which overwrites the current draft). This can be done either by the website, or the stage workflow. Important is to keep the id the same.
apply minor changes: For minor changes it is often not necessary to involve the contributor directly. Such changes could be e.g. obvious typos in some of the fields. Reviewers can apply changes locally to the resource package and re-upload (using the same id), or, given S3 credentials, files could also be changed in place. Latter way of changing requires manually triggering the tests (see Testing section).
accept: Accepting the resource draft via the web interface triggers the publish workflow, which creates a new unique resource version. As a result, the resource is published, the draft deleted and, thus, the resource is available via the bioimage.io website. The backup workflow will upload/publish the resource version to zenodo using the bioimage.io bot account (tagged with backup.bioimage.io).

[^1]: Parameters to this workflow are bioimage.io resource concept ID (id from the rdf.yaml), and Download URL of the resource package zip-file, which should contain a publicly reachable URL to a resource package .zip-file. [^2]: Parameters to this workflow are Bioimage.io resource concept ID (id from the rdf.yaml), and Published version or 'draft' (optional, usually draft).

bioimage-io / collection