Evaluation engine for bigpicture integration with grand-challenge

[pontus]

bigpicture has a use case for indirect access to images, e.g. letting an algorithm run over images that aren't allowed to be viewed by actual humans.

One idea for fulfilling that is to align with the Grand challenge project, we've also discussed doing a demonstration of this.

A first/early iteration of such a demonstrator could

• fetch available evaluations
• claim a suitable evaluation
• download the model (evaluation image) and image ("challenge submission")
• If I understand correctly, the phase essentially will be what allows for mapping to a dataset within big picture, we can just use something static, public for now.
• There will be some additional things done (modification of the image to enable it to get the data, "shimming", we ignore that for now)
• we'll also need to manage a way to expose the images to the runtime environment (kubernetes cluster). If we have a harbor or similar registry we can put them there and e.g. get added benefit of scanning (but the setup should be solid enough that we manage even with evil submission and evaluation images)
• run the submission image over the dataset by submitting jobs to the kubernetes cluster where you are running, once the submission completes the evaluation image can pass over its output (this can be done in the same pod if desirable or kept separately)
• once the dataset is passed over, results needs to be compiled, TBD
• with the results compiled, update the evaluation

There are a lot of things that explicitly do not need to happen now, e.g. later on we'll want to verify somehow that the evaluation request was created by someone who is allowed to do indirect (or possibly direct) access on the dataset, how to shim the images, and so on.

This shouldn't be exposed to any incoming requests but should still be considered very sensitive.

Likely golang is most suitable to provide small, secure and understandable code for now.

[pontus]

Some of the example data in the api explorer seems weird. I have some example code I got some time ago that can be passed, there's no guarantee that works with what ended up in the API but it can be nice to have a look at.

[minha12]

(1) The API explorer doesn't work correctly indeed. Could you please share your codes? (2) For demo purposes, can I set Bigpicture as simple as a container that runs S3 with some sample images?