search

data2health / DREAM-Challenge

EHR DREAM Challenge
7 stars 2 forks source link

Make sure that invalid submissions in UW queue are propogated to the main queue #39

Closed tschaffter closed 4 years ago

tschaffter commented 4 years ago

@thomasyu888 Is this correct that you are doing some work on tiding the submissions IDs? Can you please keep track of what you are doing here and close this ticket once the task is completed?

thomasyu888 commented 4 years ago

@tschaffter. What do you mean by tiding?

tschaffter commented 4 years ago

@thomasyu888 I meant "Tidying up" instead of "Tiding", which is still a loose title for this ticket. Here I'm trying to capture work which I think you mentioned relative to submission IDs. Does that ring a bell?

thomasyu888 commented 4 years ago

Ah. I changed the title. The workflow is

  1. Submit model to main queue,
  2. Validated models from 1 are submitted to UW internal queue
  3. Ideally models in 2 would be SCORED, but.... if not, have to make sure that the submissions made in 1 are made INVALID, or else participants won't be able to submit more. (due to the submission quota)
tschaffter commented 4 years ago

@thomasyu888 Is there an update for this issue?

thomasyu888 commented 4 years ago

There isn't. I'm manually rejecting submissions that pass the synthetic data set but fail on the main UW queue.

tschaffter commented 4 years ago

Is there a way to automatic this task?

thomasyu888 commented 4 years ago

Hmm.. will have to look into this more. One possible solution is to add a step to check the status of the UW run, and if INVALID set the successful submission to the main queue as REJECTED. However, there are cases where the rundocker step fails. In this scenario, we would also need to set the submission to the main queue as REJECTED.

Another solution is to have a tool external to the workflows to look at submissions that are VALIDATED in the main queue and INVALID in the UW queue. When this is seen, the VALIDATED submission becomes REJECTED.

The 2nd solution is definitely easier to implement, but will have some sort of lag time (1-5 minutes). The 1st solution would require more thought.

trberg commented 4 years ago

I agree, I think the 2nd solution is an easier short term fix. I've also been using the challengeutils tool to change the status.

thomasyu888 commented 4 years ago

Ya, I have been using challengeutils to update as well.

trberg commented 4 years ago

Would there be a way to use the challengeutils tool as a cronjob and just check submissions every minute or so?

thomasyu888 commented 4 years ago

That's the second solution. It would just be a script that queries the two leaderboards, and sets respective submissions to REJECTED. I can implement once I'm back from the dream conference.

thomasyu888 commented 4 years ago

Closing for now, since I implemented the workaround.