search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

Submission Pipeline fails running through 104 submissions (10 submissions at a time) #29

Closed thomasyu888 closed 5 years ago

thomasyu888 commented 5 years ago

Workflow Administrator:

Submission 9683420 failed. The workflow description is: WorkflowJobImpl [submissionId=null, containerName=workflow_job.95aa5e0d-9a4f-4c96-85d2-182b8010ad5c, container=com.github.dockerjava.api.model.Container@7a073a31[command=toil-cwl-runner --defaultMemory 100M --retryCount 0 --defaultDisk 1000000 --workDir /home/ec2-user/workflow/91a331c2-3dcd-4743-9231-df8ff398abfb --noLinkImports NCI-CPTAC-Challenge-workflowtest/workflow/docker_agent_workflow.cwl /home/ec2-user/workflow/91a331c2-3dcd-4743-9231-df8ff398abfb/TMP18122626125911529141.yaml,created=1543287756,id=fef657a3722624b05b647bf748e5ae74c05a9cf742e490fb466f4f321fb8239a,image=sagebionetworks/synapseworkflowhook-toil,imageId=sha256:228a8cbc988486142c2544bbef985d409b896e71dd9f76e1cd37348c09be108b,names={/workflow_job.95aa5e0d-9a4f-4c96-85d2-182b8010ad5c},ports={},labels={},status=Up 2 seconds,sizeRw=,sizeRootFs=,hostConfig=com.github.dockerjava.api.model.ContainerHostConfig@7171bf1b[networkMode=default],networkSettings=com.github.dockerjava.api.model.ContainerNetworkSettings@16cb831d[networks={bridge=com.github.dockerjava.api.model.ContainerNetwork@2b7ebde8[ipamConfig=,links=,aliases=,networkID=c184b234774f2ec6a090705bc131b2e4dc6ea3d3a905bc925ba5c0461f7a095a,endpointId=489b566a39433ed3d16128c8b6dd9f02f570a6d51561a4d79671fa2c2c48413d,gateway=172.17.0.1,ipAddress=172.17.0.4,ipPrefixLen=16,ipV6Gateway=,globalIPv6Address=,globalIPv6PrefixLen=0,macAddress=02:42:ac:11:00:04]}]], submissionParameters=null]

thomasyu888 commented 5 years ago

The message is:

java.lang.RuntimeException: org.sagebionetworks.client.exceptions.SynapseClientException: javax.net.ssl.SSLProtocolException: Received close_notify during handshake
at org.sagebionetworks.MessageUtils.sendMessage(MessageUtils.java:176)
at org.sagebionetworks.WorkflowHook.updateJob(WorkflowHook.java:515)
at org.sagebionetworks.WorkflowHook.updateWorkflowJobs(WorkflowHook.java:364)
at org.sagebionetworks.WorkflowHook.execute(WorkflowHook.java:187)
at org.sagebionetworks.WorkflowHook.main(WorkflowHook.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: org.sagebionetworks.client.exceptions.SynapseClientException: javax.net.ssl.SSLProtocolException: Received close_notify during handshake at org.sagebionetworks.client.BaseClientImpl.putFileToURL(BaseClientImpl.java:306) at org.sagebionetworks.client.SynapseClientImpl.putFileToURL(SynapseClientImpl.java:1519) at org.sagebionetworks.client.MultipartUpload.uploadMissingParts(MultipartUpload.java:129) at org.sagebionetworks.client.MultipartUpload.uploadFile(MultipartUpload.java:94) at org.sagebionetworks.client.SynapseClientImpl.multipartUpload(SynapseClientImpl.java:4714) at org.sagebionetworks.client.SynapseClientImpl.uploadStringToS3(SynapseClientImpl.java:2567) at org.sagebionetworks.client.SynapseClientImpl.sendStringMessage(SynapseClientImpl.java:2560) at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.sagebionetworks.client.SynapseProfileProxy$SynapseInvocationHandler.invoke(SynapseProfileProxy.java:55) at com.sun.proxy.$Proxy27.sendStringMessage(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.sagebionetworks.SynapseClientFactory$1$1.execute(SynapseClientFactory.java:46) at org.sagebionetworks.SynapseClientFactory$1$1.execute(SynapseClientFactory.java:43) at org.sagebionetworks.ExponentialBackoffRunner.execute(ExponentialBackoffRunner.java:63) at org.sagebionetworks.SynapseClientFactory$1.invoke(SynapseClientFactory.java:43) at com.sun.proxy.$Proxy27.sendStringMessage(Unknown Source) at org.sagebionetworks.MessageUtils$1.execute(MessageUtils.java:172) at org.sagebionetworks.MessageUtils$1.execute(MessageUtils.java:169) at org.sagebionetworks.ExponentialBackoffRunner.execute(ExponentialBackoffRunner.java:63) at org.sagebionetworks.MessageUtils.sendMessage(MessageUtils.java:169) ... 10 more Caused by: javax.net.ssl.SSLProtocolException: Received close_notify during handshake at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:255) at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:244) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.sagebionetworks.simpleHttpClient.SimpleHttpClientImpl.execute(SimpleHttpClientImpl.java:261) at org.sagebionetworks.simpleHttpClient.SimpleHttpClientImpl.putFile(SimpleHttpClientImpl.java:113) at org.sagebionetworks.client.BaseClientImpl.putFileToURL(BaseClientImpl.java:296) ... 33 more

thomasyu888 commented 5 years ago

My machine is m4.xlarge, running 10 submission at a time. No time quota set. Not sure how to read this error above.

brucehoff commented 5 years ago

@thomasyu888 The error is some sort of transient connection error. The fix is for me to include this case in the retry logic: https://github.com/Sage-Bionetworks/SynapseWorkflowHook/commit/61192a9054b65f7e9db971a66279159e6dd125f7