Improvements to Reviewing Process in bioimageio-uploader

[cfusterbarcelo]

Description: Following recent discussions and feedback on the current reviewing process (as in the last HT hackathon and Meeting Minutes), a few challenges and potential improvements have been identified that could streamline the workflow, increase security, and enhance the usability of the bioimageio-uploader. Below is a summary of the issues raised and suggestions for improvement.

Problems Identified:

1. Model Review via S3 Bucket:
   • Currently, the model reviewing process relies heavily on mounting an S3 bucket and manually browsing for models, which can be inefficient.
   • There is only one shared credential for all users, with extensive permissions on the bucket. This poses a security risk as accidental or unauthorized actions could affect the entire bucket.
2. Error Log Accessibility:
   • Reading and interpreting error logs on the uploader can be challenging, especially for reviewers trying to diagnose issues.
3. Requirement of Sample Input/Output for Testing:
   • While requiring the model to produce exact outputs from sample inputs has been beneficial for reproducibility and confidence in the models, it could present hardware compatibility issues for some users.
4. Frequent Error - "Reproduce test outputs from test inputs":
   • Several models seem to encounter a recurring error related to reproducing expected outputs from test inputs. Clearer guidelines on diagnosing and troubleshooting this error would be beneficial.

Suggested Improvements:

1. Add Contributor Information in Model Metadata:
   • Including the primary contributor’s name directly within the model metadata would help reviewers quickly identify the original creator of the model.
2. Direct Link or Access to Model Download:
   • Providing a direct link or simplified access to download the entire model package could streamline the reviewing process and make it more efficient.
3. Establish a Conda Environment for Testing:
   • Providing a conda environment file or a standard testing environment setup within the uploader repository could make it easier for reviewers to reproduce model tests and potentially avoid hardware-related discrepancies.
4. Clear Steps for Troubleshooting Common Errors:
   • For common errors like the “reproduce test outputs from test inputs” issue, including step-by-step troubleshooting guidance would help reviewers resolve this error more effectively.

With this, we hope to improve and make more reviewer friendly the process for reviewing uploaded models. Kudos to @mese79 and @veegalinova for their help.

[oeway]

@FynnBe There are some issue with the model pending uploads page:

• The date is wrong
• The authors displayed should be the uploader, not the original authors

Could you please update the CI so it produce a collection draft which each item contains the correct info.status.timestamp, and also the status.description will display the uploader?

                <p>⏳ {info.status.name} ({info.status.step}/{info.status.num_steps})</p>
                <p>{info.status.description}</p>
                <p>{new Date(info.status.timestamp).toString()}</p>
                <a href={info.status.run_url} target="_blank"><img src="{github}" alt="github icon">Github CI Logs</a>

cc @cfusterbarcelo

[FynnBe]

not all problems/improvements are adressed yet