Create queues and dashboards to support 3 submission types

[Simon-Harris-IBM]

At some point we need to create a consolidated leaderboard containing the sorted submission results.

MIT/IBM may want 3 leaderboards:

1. Systems that take as input the subset of ObjectNet that intersects with ImageNet using ImageNet as the training set.
2. Systems that take as input the subset of ObjectNet that intersects with ImageNet using any data as training set.
3. Systems that take as input all of ObjectNet using any data as the training set.

Not sure if it will be possible to seperate out submissions in this way.

[Simon-Harris-IBM]

Will need to create 3 new front-end fan-out queues - one for each of these leaderboards....

[thomasyu888]

The workflow now supports only needing one queue.

[Simon-Harris-IBM]

@mummert-ibm Will we require changes to validate_and_score.py to support these different types of submissions ???

[thomasyu888]

This will be a difficult concept to capture. We would have to leave this entirely up to the participant to submit this information and trust that they are providing it correctly.

[Simon-Harris-IBM]

Agreed. This will have to be left up to the participants honesty. Even if they do submit a model pre-trained on data to a non-matching queue, and we have the model, it may still be difficult to determine exactly what data the model was trained on.

I'll double check with MIT this is what they want.

[Simon-Harris-IBM]

@thomasyu888 what would be the best way to implement this if we need to pass a flag/arg onto validate_and_score.py ? We have 3 seperate queues - we could link them to 3 slightly different versions of main_queue_wf.cwl which pass in the argument to validate_and_score.py. But this seems like a v. inefficient way of doing this ?

[thomasyu888]

@Simon-Harris-IBM, Ideally, this information could be submitted upon time of submission but currently, we only have the submission name that participants can use to submit this sort of information.

That being said, if we had three different queues, we could have a CWL mapping tool that maps evaluation queue id to the type so we wouldn't need three workflows.

[Simon-Harris-IBM]

I've created 3 different q's so i think the use of a CWL mapping tool to eval q would be v. helpful :-)

[thomasyu888]

@Simon-Harris-IBM, so the idea is that on one smaller instance, you would listen to these 3 different queues and submit to the n internal queues available (n number of instances) Is that correct? You can effectively run the main workflow on these 3 queues.

How different would the validation and scoring be?

[Simon-Harris-IBM]

@thomasyu888 Thats correct. We have a non-GPU machine that is currently just monitoring one submission queue and performing the fanout to the backends. The idea is to extend this such that it monitors all 3 queues, and fans-out to the same set of backend queues.

I added a comment above to find out from @mummert-ibm how different the scoring would need to be (if at all)!

[Climbgunks]

David would have to provide guidance here.

Two of the queues are only returning ImageNet labels, they could use the same dataset and answers. (roughly 10k * 113/ 313 images)

The other queue is the full ObjectNet dataset (~10k images)

I don't think the scoring program has to change... just which image directory we map to the queues and the answer file.

[Simon-Harris-IBM]

Following queues (and associated files) have been setup on synapse ObjectNet project:

• ObjectNet Challenge sub-challenge 1 submission queue
• ObjectNet Challenge sub-challenge 2 submission queue
• ObjectNet Challenge sub-challenge 3 submission queue

Have not linked these into the backend workflow orchestrator yet though.