Create equivelent of `collection.json` that lists all staged e.g. `staged_collections.json`

[jmetz]

Part of https://github.com/bioimage-io/bioimageio-uploader-service/issues/9 (which in turn is part of meta-issue https://github.com/bioimage-io/bioimageio-uploader/issues/65 ) requires that we get all staged versions.

As this involves doing a list operation on S3 followed by collating all the individual versions.json files, and in principle this will happen many times between each new stage, I would recommend that we build this in backoffice when stage and publish are called, similar to the way that collection.json is currently generated.

What do you think @oeway + @FynnBe ?

The alternative, to have the hypha service dynamically generate the same JSON response every time the status page is loaded, seems very inefficient.

[FynnBe]

yeah, I thought of a collection_staged.json before.. (note the different name! 😕) but yeah... I can add that to backoffice 👍

[jmetz]

Even better - collection_staged.json probably makes more sense.

I think it should probably be updated on each stage as well as publish call, right?

Presumably more or less a copy/paste of the code you already wrote to create collection.json, except that you'll pick out staged instead of published versions.

[FynnBe]

publish is not changing the staged versions (well, their status, but that is not included in the collection.json) I added the dois to the collection.json files which are updated by backup. so after backup both collections are regenerated. and yes after stage staged collection and after publish the regular/published collection... impl in 23ec0fb88b804718e42ebe10e91a4f19a5b25cd1

[FynnBe]

(sorry, developing this on main due to changes to gh wfs... ) 🤫