Can default values be provided for 'leaderboard' widgets

[Simon-Harris-IBM]

I'm looking to improve the info on the submission dashboard the participants use to track their data: https://www.synapse.org/#!Synapse:syn21445381/wiki/601621

I'm thinking we could remove "status" and "workflow status" from the leaderboard if we can provide a default value for "Prediction file status". Then, when "prediction file status" is blank (because the scoring has not been completed), we can simply supply a default value of "In progress". When prediction is complete, we'll take the value from the results.json file.

Looks like default values are possible in "Table" widgets: https://user-guides.synapse.org/articles/tables.html . What about "leaderboard" widgets ?

[thomasyu888]

@Simon-Harris-IBM This may be a duplicate of #13 . We can most certainly set that annotation to be "IN PROGRESS" at the very beginning when users first submit their model. However, there won't be any notion that the submission is actually running to the user aside from that annotation.

Unfortunately, Tables are different from leaderboards, so there are no default values in "Table" widgets

[Simon-Harris-IBM]

Thanks @thomasyu888 . I agree this may be a dup of #13 (which I haven't tried running as a cron job yet). Was just thinking a simpler solution could be to use default values ?

[thomasyu888]

@Simon-Harris-IBM Ah, no theres no default values. Also the #7 is different from #13. There is a slight difference based on these couple of things.

• The submission "status" such as RECEIVED, ACCEPTED... are used in the number of times a participant can submit.
• We needed the main queue submission to be updated of the progress (status) of how its linking internal submission is running.

7 is to make sure all invalid submissions will be propagated back to the main queue to reset them to an INVALID state so that participants can submit more

Part of #13 can most certainly do this, but if the workflow fails in anyway during the runtime, a submission will be INVALID. in this case, its entirely possible that the submission has not gotten to the step where we update the status of the main queue submission from ACCEPTED -> INVALID. Therefore #7 is important.

[Simon-Harris-IBM]

Thanks @thomasyu888 So can I just clarify what is supposed to currently happen ?

The following dashboard is meant to represent what participants will see from the main 'ObjectNet Challenge': https://www.synapse.org/#!Synapse:syn21445381/wiki/601621

When I submit a job I usually see 'Status=Accepted', 'Workflow status=DONE' & 'Prediction file status=' once the job has been passed onto the backend Q. When the job is complete on the backend Q I see 'Status=Accepted', 'Workflow status=DONE' & 'Prediction file status=VALIDATED'.

So participants don't get a good feel as to the status of their job. I was hoping that one of the status fields would be dynamically updated as the job on the backend was being executed ??

[thomasyu888]

@Simon-Harris-IBM . No problem. Currently all I have done with #13 is to update the main queue with the scores from the internal queues.

I was going to wait for your return to discuss more about how you want to track the submission because there are many ways we can. Status=Accepted and Workflow status=Done both only pertain to the status of the submission on the main queue, so those two are essentially useless. There are a few possibilities. We do something where we describe the meanings of the statues:

Its important we keep "status" because that shows the RECEVIED state, but by default prediction file status will be blank.

• RECEIVED: Your submission has been received
• EVALUATION_IN_PROGRESS: Determining internal queue to submit to and validating docker image
• ACCEPTED: Submitted to internal queues for submissions
• OPEN: Submission is running --The reason the "status" variable cannot be EVALUATION_IN_PROGRESS after the submission has been completed in the main queue is because that annotation will make the orchestrator believe there is a submission running on the instance causing the orchestrator to fail. Hence (OPEN)
• VALIDATED: prediction file generated is valid
• SCORED: prediction file has been scored
• INVALID: Invalid model

As I was typing this, I realized that I actually like your method better. If we just use the prediction file status column, there are the statues and what they would mean:

• (blank): Submission has been received but not being executed
• EVALUATION_IN_PROGRESS: submission is currently being submitted to internal queue and executing on internal queue
• VALIDATED: same as above
• SCORED: same as above
• INVALID: same as above

That being said, it is important to note that neither of these solutions will tell participants if their models are actually running in the internal queues (aside from their log files)

[Simon-Harris-IBM]

Looks good @thomasyu888 though not sure we will need to distinguish between validated and scored in our case..... Can we change the tag in anyway, so instead of "SUBMITTED_FOR_EVALUATION" can we use "EVALUATION_IN_PROGRESS" instead. It's less definitive and we can document that the job miught not actually be running.... Also, is it possible to have prediction file status of 'RECEIVED' instead of blank.... Thanks...

[thomasyu888]

@Simon-Harris-IBM unfortunately there isn't. The prediction file status is something that is added to the submission once the workflow starts so it will just be blank. We can definitely use EVALUATION_IN_PROGRESS for the prediction file status column.

[Simon-Harris-IBM]

@thomasyu888 I think that's a reasonable compromise and much better than what we have at the moment..... We'll explain in the wiki that blank means 'received but not yet executing'....

[Simon-Harris-IBM]

@thomasyu888 #33 now shows a status of "EVALUATION_IN_PROGRESS" on the main submission dashboard when an evaluation is in progress. There's one issue though. We have 3 backend queues/servers so I submitted 4 jobs to see what the status of the 4th job would be whilst it waited. Unfortunately, whilst the 4th job is waiting and has not been allocated to backend q, its status is also "EVALUATION_IN_PROGRESS" -- which is a little misleading.

[Simon-Harris-IBM]

Would this issue be resolved if we just move the call to "create_status_annotation.cwl" until after "get_backend_queue" step in the workflow ?

[thomasyu888]

@Simon-Harris-IBM Not exactly... CWL doesn't follow order. You would have to move it into the docker workflow. That is why we added previous_annotation_finished, to force the steps to be executed in the order we want them to be.

[Simon-Harris-IBM]

@thomasyu888 Can you take a look at the following commits please:

• 0c2fba76cc08a090156adee27bf995e975aa3a3b
• 29d83337360638936631aa1435b59f7308e84063

I moved creation of update_status.json file into the submit_to_challenge.cwl file and moved annotate_status.cwl to after the call to submit_to_challenge.cwl in the main_queue_wf.cwl. This seems to work in that the 4th job queued had a status until one of the occupied queues freed up. At which point, the job was queued and the status changed to "EVALUATION_IN_PROGRESS".

If you are ok with this change I'll close this issue.

[thomasyu888]

@Simon-Harris-IBM Do not raise value error here: https://github.com/Simon-Harris-IBM/ObjectNetChallenge-Workflows/blob/master/submit_to_challenge.cwl#L71 Or the workflow will quit before you can set the status.

You can update this step: https://github.com/Simon-Harris-IBM/ObjectNetChallenge-Workflows/blob/master/main_queue_wf.cwl#L148 with the below.

  check_status:
    run: https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.1/check_status.cwl
    in:
      - id: status
        source: "#validate_docker/status"
      - id: previous_annotation_finished
        source: "#annotate_status/finished"
    out: [finished]

This way the check status step only runs after you run submit_to_challenge and that annotation step.

[Simon-Harris-IBM]

@thomasyu888 your suggestion above worked great..... Closing this issue.