I would hold off on this PR until you test if the orchestrator can just be linked to the same queue on multiple machines I will describe the tools and updates here. In this update:
docker_agent_workflow.cwl: the queue you have been using will no longer take direct submissions from participants, submissions will be done by a service account and this workflow will take a submission that is in form of json:
{'submissionid': '11111'}
get_linked_submissionid.cwl: This tool simply get the above json and extracts out the actual submission id that the participant submitted
get_submission_docker.cwl: This is a tool that obtains the docker submission, It is unique because it adds the submitter and user Id of the actual participant so this information can be added to the leaderboard
main_queue_wf.cwl : This will be the main queue that the participants see, They will submit to this queue and it will generate a submission on their behalf to one of the queues. Missing: The tool that selects which queue to submit to, right now it just defaults to one queue.
submit_to_challenge.cwl: This tool takes the submission to the main queue and write a json described above and submits that json file to the chosen internal evaluation queue.
I would hold off on this PR until you test if the orchestrator can just be linked to the same queue on multiple machines I will describe the tools and updates here. In this update:
docker_agent_workflow.cwl: the queue you have been using will no longer take direct submissions from participants, submissions will be done by a service account and this workflow will take a submission that is in form of json:{'submissionid': '11111'}get_linked_submissionid.cwl: This tool simply get the above json and extracts out the actual submission id that the participant submittedget_submission_docker.cwl: This is a tool that obtains the docker submission, It is unique because it adds the submitter and user Id of the actual participant so this information can be added to the leaderboardmain_queue_wf.cwl: This will be the main queue that the participants see, They will submit to this queue and it will generate a submission on their behalf to one of the queues. Missing: The tool that selects which queue to submit to, right now it just defaults to one queue.submit_to_challenge.cwl: This tool takes the submission to the main queue and write a json described above and submits that json file to the chosen internal evaluation queue.