search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

Hook stuck with no error message #33

Closed thomasyu888 closed 5 years ago

thomasyu888 commented 5 years ago

The Workflow hook times out with no error message and I have to docker-compose down and docker-compose up to make it work.

I am running 2 concurrent docker submissions at a time, and the timeout happens around ~ 55 submissions.

What could be causing this issue?

thomasyu888 commented 5 years ago 
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 2, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null

It is stuck on this message

brucehoff commented 5 years ago

@thomasyu888 You use the phrase 'time out' and then you say it's 'stuck'. It's not clear if the hook is still running. Can you post the output of 'docker ps -a' when the hook is in this state? I'm curious (1) if the hook is still running and (2) how many submissions are running. It might be simply that two submissions are running and the 'hook' is waiting for one to finish before starting a new one.

thomasyu888 commented 5 years ago

Sorry for the confusion. I meant that the hook gets stuck at that line and it continues to run without updating submission statueses and running new submissions.

I will reproduce this issue, but basically only the hook is running while everything else has finished running. Only upon stopping and starting the hook does it update the submission status and start new jobs.

brucehoff commented 5 years ago

@thomasyu888 Can you {code} docker logs --tail 200 synapseworkflowhook_workflow-hook_1 {code} (or whatever the container name is for the hook) and attach the output to this issue? Also, when the system is in the problematic state attach to output of: {code} docker ps -a {code} to show whether the hook container is still running or has stopped.

thomasyu888 commented 5 years ago

I will once I reproduce the issue. But I can tell you this, the only thing that is running would be the synapseworkflowhook_workflow-hook_1... and its logs are what I posted above, but repeated... it just gets stuck on the last line. (doesnt timeout, sorry, timeout gives it the impression that the hook stopped running)

....
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 2, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 2, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 2, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null

The one thing i do notice though is that for whatever reason, the workflow_job.4f2ecca5-761d... is run to completion, but is in an exited state. I will reproduce and get you those logs

thomasyu888 commented 5 years ago

I think I may have resolved this issue by setting a memory limit on each docker run command i do for submissions. I will double check.

thomasyu888 commented 5 years ago 
[ec2-user@ip-10-23-51-161 ~]$ docker ps -a
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS                      PORTS               NAMES
059d1810d45a        sagebionetworks/synapseworkflowhook-toil                     "toil-cwl-runner --d…"   12 hours ago        Exited (1) 12 hours ago                         workflow_job.c95a96c4-2da8-47bd-af8b-80330c14bc70
cf582b6fbd9a        sagebionetworks/synapseworkflowhook-toil                     "toil-cwl-runner --d…"   12 hours ago        Exited (1) 12 hours ago                         workflow_job.74d99a13-5b76-40b1-bd4c-877a70d4e961
3a05626d6f27        sagebionetworks/synapseworkflowhook-toil                     "toil-cwl-runner --d…"   12 hours ago        Exited (1) 12 hours ago                         workflow_job.b92c7e34-e8dd-48b5-8613-0d3c8bc8f390
063dc20aba6a        sagebionetworks/synapseworkflowhook-toil                     "toil-cwl-runner --d…"   12 hours ago        Exited (0) 12 hours ago                         workflow_job.0ee90753-7560-4748-bb15-a38ef0265973
bf04f62912f2        docker.synapse.org/syn8228304/archive-9654231-sc2            "/score_sc2.sh"          12 hours ago        Exited (0) 12 hours ago                         9683417
316d0f02a841        docker.synapse.org/syn8228304/archive-9653993-sc2-ov-frepo   "/score_sc2.sh"          12 hours ago        Exited (137) 12 hours ago                       9683413
f400fb00f549        docker.synapse.org/syn8228304/archive-9653910-my-repo        "/score_sc2.sh"          12 hours ago        Created                                         9683412
cba0ad386c8d        docker.synapse.org/syn8228304/archive-9652989-sub2_v6        "/score_sc2.sh"          13 hours ago        Exited (0) 12 hours ago                         9683406
624dadc0ff2d        docker.synapse.org/syn8228304/archive-9650776-sc2            "/score_sc2.sh"          33 hours ago        Created                                         9683370
e979c13b1062        sagebionetworks/synapseworkflowhook                          "/usr/local/bin/mvn-…"   33 hours ago        Up 14 hours                                     synapseworkflowhook_workflow-hook_1_8ffa8a3e644a
thomasyu888 commented 5 years ago 
[ec2-user@ip-10-23-51-161 ~]$ docker logs --tail 200 synapseworkflowhook_workflow-hook_1_8ffa8a3e644a
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683801_logs.zip already exists with a parentId: syn17088119
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Archived /tmp/9683801_logs.txt to syn17088126
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - Top level loop: checking progress or starting new job.
21:06:15.714 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:15 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn17055973
21:06:15.989 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:15 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056007
21:06:16.172 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:16 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn17055973
21:06:16.424 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:16 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056009
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f, --noLinkImports, NCI-CPTAC-Challenge-workflowtest/workflow/docker_agent_workflow.cwl, /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f/TMP12994526190779200932.yaml]
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683805 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683806 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683807 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683808 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683809 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683810 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683811 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683812 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683813 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683814 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683815 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683816 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683837 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683838 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683839 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683840 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683841 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683842 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683843 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683844 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683845 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683846 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683847 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683848 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683849 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683850 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683851 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683852 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683853 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683854 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683855 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683856 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683874 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683877 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683878 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683880 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683881 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683882 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683914 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683915 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683916 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683917 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683918 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683919 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683920 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683921 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683922 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683923 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683924 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683925 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683926 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683927 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683928 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683929 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683930 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683932 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683933 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683936 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683937 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683939 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683940 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683943 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683944 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683946 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683947 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683948 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683950 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683951 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683952 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683956 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683957 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683958 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683959 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683960 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683961 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683962 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683963 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683964 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683965 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683966 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683968 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683969 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683972 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683973 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683974 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683982 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683983 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683984 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683985 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683986 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683987 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683988 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683989 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683990 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683991 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683992 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683993 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683994 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683995 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683996 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683997 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683998 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683999 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684000 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684001 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684120 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684121 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684122 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684123 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684124 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684125 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684126 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684127 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684128 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684129 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684130 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684131 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684132 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684133 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684134 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684135 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684136 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684137 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684138 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684139 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684140 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684141 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684142 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684143 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684144 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684145 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684146 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684147 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684148 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684149 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684150 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684151 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684152 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684153 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684154 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684155 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684156 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684157 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684158 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684159 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684160 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684161 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684162 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684163 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684164 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684166 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684167 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684168 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684169 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684170 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684171 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684172 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684173 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684174 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684175 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684195 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684197 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684198 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684199 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684200 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684201 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684202 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Found 273 bytes to log.
21:06:23.667 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:23 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn17055973
Dec 22, 2018 9:06:24 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056009
21:06:24.001 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Will archive /tmp/9683804_logs.txt
21:06:24.932 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:24 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804_logs.zip already exists with a parentId: syn17088155
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Archived /tmp/9683804_logs.txt to syn17088166
[ec2-user@ip-10-23-51-161 ~]$ docker logs --tail 200 synapseworkflowhook_workflow-hook_1_8ffa8a3e644a
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683801_logs.zip already exists with a parentId: syn17088119
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Archived /tmp/9683801_logs.txt to syn17088126
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - Top level loop: checking progress or starting new job.
21:06:15.714 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:15 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn17055973
21:06:15.989 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:15 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056007
21:06:16.172 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:16 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn17055973
21:06:16.424 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:16 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056009
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f, --noLinkImports, NCI-CPTAC-Challenge-workflowtest/workflow/docker_agent_workflow.cwl, /home/ec2-user/workflow/34ccc78a-70ce-4226-bf3a-34de851b029f/TMP12994526190779200932.yaml]
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683805 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683806 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683807 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683808 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683809 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683810 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683811 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683812 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683813 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683814 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683815 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683816 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683837 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683838 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683839 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683840 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683841 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683842 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683843 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683844 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683845 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683846 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683847 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683848 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683849 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683850 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683851 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683852 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683853 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683854 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683855 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683856 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683874 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683877 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683878 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683880 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683881 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683882 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683914 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683915 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683916 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683917 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683918 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683919 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683920 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683921 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683922 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683923 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683924 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683925 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683926 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683927 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683928 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683929 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683930 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683932 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683933 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683936 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683937 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683939 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683940 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683943 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683944 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683946 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683947 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683948 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683950 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683951 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683952 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683956 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683957 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683958 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683959 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683960 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683961 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683962 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683963 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683964 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683965 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683966 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683968 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683969 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683972 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683973 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683974 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683982 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683983 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683984 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683985 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683986 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683987 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683988 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683989 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683990 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683991 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683992 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683993 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683994 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683995 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683996 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683997 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683998 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9683999 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684000 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684001 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684120 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684121 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684122 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684123 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684124 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684125 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684126 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684127 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684128 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684129 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684130 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684131 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684132 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684133 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684134 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684135 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684136 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684137 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684138 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684139 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684140 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684141 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684142 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684143 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684144 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684145 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684146 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684147 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684148 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684149 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684150 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684151 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684152 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684153 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684154 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684155 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684156 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684157 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684158 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684159 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684160 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684161 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684162 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684163 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684164 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684166 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684167 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684168 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684169 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684170 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684171 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684172 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684173 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684174 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684175 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684195 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684197 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684198 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684199 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684200 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684201 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684202 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - We have met or exceeded the maximum concurrent workflow count, 4, so we will not start 9684203 at this time.
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - PROGRESS: null
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Found 273 bytes to log.
21:06:23.667 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:23 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn17055973
Dec 22, 2018 9:06:24 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804 already exists with a parentId: syn17056009
21:06:24.001 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Will archive /tmp/9683804_logs.txt
21:06:24.932 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
Dec 22, 2018 9:06:24 PM org.sagebionetworks.ExponentialBackoffRunner execute
SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9683804_logs.zip already exists with a parentId: syn17088155
[org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.Archiver - Archived /tmp/9683804_logs.txt to syn17088166
thomasyu888 commented 5 years ago

Please refer to #34