search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

No corresponding open Synapse submission error while running multiple submissions #14

Closed thomasyu888 closed 6 years ago

thomasyu888 commented 6 years ago 
WARNING: The SHARE_RESULTS_IMMEDIATELY variable is not set. Defaulting to a blank string.
WARNING: The DATA_UNLOCK_SYNAPSE_PRINCIPAL_ID variable is not set. Defaulting to a blank string.
Recreating synapseworkflowhook_workflow-hook_1 ... done
Attaching to synapseworkflowhook_workflow-hook_1
workflow-hook_1  | [INFO] Scanning for projects...
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.pom
Downloaded from central: http://repo1.maven.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.pom (7.7 kB at 47 kB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/codehaus/mojo/mojo-parent/28/mojo-parent-28.pom
Downloaded from central: http://repo1.maven.org/maven2/org/codehaus/mojo/mojo-parent/28/mojo-parent-28.pom (26 kB at 1.0 MB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/codehaus/codehaus-parent/3/codehaus-parent-3.pom
Downloaded from central: http://repo1.maven.org/maven2/org/codehaus/codehaus-parent/3/codehaus-parent-3.pom (4.1 kB at 274 kB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.jar
Downloaded from central: http://repo1.maven.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.2.1/exec-maven-plugin-1.2.1.jar (38 kB at 1.2 MB/s)
workflow-hook_1  | [INFO] 
workflow-hook_1  | [INFO] ------------------< org.sagebionetworks:WorkflowHook >------------------
workflow-hook_1  | [INFO] Building WorkflowHook 1.0-SNAPSHOT
workflow-hook_1  | [INFO] --------------------------------[ jar ]---------------------------------
workflow-hook_1  | [INFO] 
workflow-hook_1  | [INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ WorkflowHook >>>
workflow-hook_1  | [INFO] 
workflow-hook_1  | [INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ WorkflowHook <<<
workflow-hook_1  | [INFO] 
workflow-hook_1  | [INFO] 
workflow-hook_1  | [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ WorkflowHook ---
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0/maven-plugin-api-2.0.pom
Downloaded from central: http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0/maven-plugin-api-2.0.pom (601 B at 32 kB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/apache/maven/maven/2.0/maven-2.0.pom
Downloaded from central: http://repo1.maven.org/maven2/org/apache/maven/maven/2.0/maven-2.0.pom (8.8 kB at 516 kB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.pom
Downloaded from central: http://repo1.maven.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.pom (11 kB at 623 kB/s)
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-9/plexus-container-default-1.0-alpha-9.jar
workflow-hook_1  | Downloading from central: http://repo1.maven.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar
Downloaded from central: http://repo1.maven.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar (53 kB at 510 kB/s)
workflow-hook_1  | Downloaded from central: http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-9/plexus-container-default-1.0-alpha-9.jar (195 kB at 1.8 MB/s)
workflow-hook_1  | ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - Precheck completed successfully.
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WorkflowHook - Top level loop: checking progress or starting new job.
workflow-hook_1  | 18:48:09.400 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:48:09 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn16809914
workflow-hook_1  | 18:48:10.537 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:48:10 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn16809914
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /Users/ThomasY/sandbox/33546a2f-52f8-4ef0-a549-1354a29fab2f
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /Users/ThomasY/sandbox/33546a2f-52f8-4ef0-a549-1354a29fab2f, --noLinkImports, ChallengeWorkflowTemplates-master/scoringHarness_workflow.cwl, /Users/ThomasY/sandbox/33546a2f-52f8-4ef0-a549-1354a29fab2f/TMP4672399095637911998.yaml]
workflow-hook_1  | 18:49:13.511 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:13 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn16809914
workflow-hook_1  | 18:49:14.075 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:14 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn16809914
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /Users/ThomasY/sandbox/f80356c2-b5c5-418c-a28f-663d95a446ec
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /Users/ThomasY/sandbox/f80356c2-b5c5-418c-a28f-663d95a446ec, --noLinkImports, ChallengeWorkflowTemplates-master/scoringHarness_workflow.cwl, /Users/ThomasY/sandbox/f80356c2-b5c5-418c-a28f-663d95a446ec/TMP13686561282523269224.yaml]
workflow-hook_1  | 18:49:17.573 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:17 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn16809914
workflow-hook_1  | 18:49:18.535 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:18 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn16809914
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /Users/ThomasY/sandbox/dd0442a9-bcd6-49be-b15a-84d29ec22328
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /Users/ThomasY/sandbox/dd0442a9-bcd6-49be-b15a-84d29ec22328, --noLinkImports, ChallengeWorkflowTemplates-master/scoringHarness_workflow.cwl, /Users/ThomasY/sandbox/dd0442a9-bcd6-49be-b15a-84d29ec22328/TMP16701659416525708586.yaml]
workflow-hook_1  | 18:49:22.472 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:22 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn16809914
workflow-hook_1  | 18:49:23.569 [org.sagebionetworks.WorkflowHook.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-hook_1  | Oct 04, 2018 6:49:23 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-hook_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn16809914
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - Mounting: /var/run
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - workingDir: /Users/ThomasY/sandbox/a9eafcc0-c448-4796-998b-9427549906cf
workflow-hook_1  | [org.sagebionetworks.WorkflowHook.main()] INFO org.sagebionetworks.WES - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /Users/ThomasY/sandbox/a9eafcc0-c448-4796-998b-9427549906cf, --noLinkImports, ChallengeWorkflowTemplates-master/scoringHarness_workflow.cwl, /Users/ThomasY/sandbox/a9eafcc0-c448-4796-998b-9427549906cf/TMP75456837548125242.yaml]
workflow-hook_1  | [WARNING] 
workflow-hook_1  | java.lang.reflect.InvocationTargetException
workflow-hook_1  |     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
workflow-hook_1  |     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
workflow-hook_1  |     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
workflow-hook_1  |     at java.lang.reflect.Method.invoke (Method.java:566)
workflow-hook_1  |     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
workflow-hook_1  |     at java.lang.Thread.run (Thread.java:834)
workflow-hook_1  | Caused by: java.lang.IllegalStateException: The following workflow job(s) are running but have no corresponding open Synapse submissions.
workflow-hook_1  |  workflow_job.8c0a758f-55e4-48a7-9d11-46c3d68cea57
workflow-hook_1  |  workflow_job.a787765e-4522-42df-8c5b-94a14c302fe6
workflow-hook_1  | One way to recover is to delete the workflow job(s).
workflow-hook_1  |     at org.sagebionetworks.WorkflowHook.updateWorkflowJobs (WorkflowHook.java:314)
workflow-hook_1  |     at org.sagebionetworks.WorkflowHook.execute (WorkflowHook.java:185)
workflow-hook_1  |     at org.sagebionetworks.WorkflowHook.main (WorkflowHook.java:103)
workflow-hook_1  |     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
workflow-hook_1  |     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
workflow-hook_1  |     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
workflow-hook_1  |     at java.lang.reflect.Method.invoke (Method.java:566)
workflow-hook_1  |     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
workflow-hook_1  |     at java.lang.Thread.run (Thread.java:834)
workflow-hook_1  | [INFO] ------------------------------------------------------------------------
workflow-hook_1  | [INFO] BUILD FAILURE
workflow-hook_1  | [INFO] ------------------------------------------------------------------------
workflow-hook_1  | [INFO] Total time: 01:28 min
workflow-hook_1  | [INFO] Finished at: 2018-10-04T18:49:31Z
workflow-hook_1  | [INFO] ------------------------------------------------------------------------
workflow-hook_1  | [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project WorkflowHook: An exception occured while executing the Java class. null: InvocationTargetException: The following workflow job(s) are running but have no corresponding open Synapse submissions.
workflow-hook_1  | [ERROR]  workflow_job.8c0a758f-55e4-48a7-9d11-46c3d68cea57
workflow-hook_1  | [ERROR]  workflow_job.a787765e-4522-42df-8c5b-94a14c302fe6
workflow-hook_1  | [ERROR] One way to recover is to delete the workflow job(s).
workflow-hook_1  | [ERROR] -> [Help 1]
workflow-hook_1  | [ERROR] 
workflow-hook_1  | [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
workflow-hook_1  | [ERROR] Re-run Maven using the -X switch to enable full debug logging.
workflow-hook_1  | [ERROR] 
workflow-hook_1  | [ERROR] For more information about the errors and possible solutions, please read the following articles:
workflow-hook_1  | [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
workflow-hook_1  | [Thread-1] INFO org.sagebionetworks.ShutdownHook - Shut down signal received.
workflow-hook_1  | [Thread-1] INFO org.sagebionetworks.ShutdownHook - Shut down complete.
synapseworkflowhook_workflow-hook_1 exited with code 1
thomasyu888 commented 6 years ago

Ahh. This error happened because I had old docker containers that weren't cleaned up. Works great!