search

tercen / flowjo_plugin

FlowJo plugin that connects with Tercen
0 stars 0 forks source link

Gateway time-out #27

Closed ginberg closed 2 years ago

ginberg commented 3 years ago

Sometimes when uploading big files a gateway time-out happens. I found out when using the dataset from https://github.com/tercen/flowjo_plugin/issues/22

This is the logging and stacktrace

2021-10-26 13:57:06,264 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 85 %
2021-10-26 13:57:06,491 DEBUG com.tercen.flowjo.UploadProgressTask  - create task
2021-10-26 13:57:06,619 DEBUG com.tercen.flowjo.UploadProgressTask  - run task
2021-10-26 13:57:06,692 DEBUG com.tercen.flowjo.UploadProgressTask  - waitDone
content-length 167
content-type text/html
date Tue, 26 Oct 2021 11:58:06 GMT
server nginx/1.19.2
strict-transport-security max-age=15724800; includeSubDomains
ServiceError(statusCode : -42 -- error : unknown -- reason : ServiceError(statusCode : 504 -- error : unknown -- reason : <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.19.2</center>
</body>
</html>
))
    at com.tercen.service.HttpClientService.onError(HttpClientService.java:53)
    at com.tercen.client.base.TaskServiceBase.waitDone(TaskServiceBase.java:112)
    at com.tercen.flowjo.UploadProgressTask.uploadFile(UploadProgressTask.java:123)
    at com.tercen.flowjo.Utils.uploadCsvFile(Utils.java:53)
    at com.tercen.flowjo.Tercen.invokeAlgorithm(Tercen.java:196)
    at com.treestar.flowjo.engine.ExternalEngine.unsynchedSubmitQuery(Unknown Source)
    at com.treestar.flowjo.engine.ExternalEngine.submitQuery(Unknown Source)
    at com.treestar.flowjo.engine.CachingEngine.unsynchedSubmitQuery(Unknown Source)
    at com.treestar.flowjo.engine.ConcurrentCachingEngine.submitQuery(Unknown Source)
    at com.treestar.flowjo.engine.Query.executeQuery(Unknown Source)
    at com.treestar.flowjo.engine.Query.executeQuery(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode.executeQueryAndProcess(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode.access$100(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode$1.run(Unknown Source)
    at com.treestar.lib.util.STaskExecutor$2.run(Unknown Source)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.base/java.lang.Thread.run(Thread.java:844)
ginberg commented 3 years ago

I think we need to increase the default NGINX timeout https://ubiq.co/tech-blog/increase-request-timeout-nginx/

ginberg commented 3 years ago

There is still a connection issue when using the websockets.

2021-11-01 14:41:26,820 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 7 %
2021-11-01 14:41:35,383 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 15 %
2021-11-01 14:41:43,373 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 23 %
2021-11-01 14:41:51,480 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 30 %
2021-11-01 14:42:31,296 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 38 %
2021-11-01 14:42:39,442 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 46 %
2021-11-01 14:42:47,636 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 53 %
2021-11-01 14:42:56,189 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 61 %
2021-11-01 14:43:04,357 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 69 %
2021-11-01 14:43:17,457 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 76 %
2021-11-01 14:43:25,574 DEBUG com.tercen.flowjo.UploadProgressTask  - file upload progress: 84 %
2021-11-01 14:43:25,906 DEBUG com.tercen.flowjo.UploadProgressTask  - create task
2021-11-01 14:43:26,275 DEBUG com.tercen.flowjo.UploadProgressTask  - Connect to: wss://stage.tercen.com/api/v1/evt/listenTaskChannel?params=%7B%22taskId%22%3A%22775278279284adeedf6c53dcb02ff2db%22%2C%22start%22%3Atrue%7D
2021-11-01 14:43:26,398 DEBUG com.tercen.flowjo.TercenWebSocketListener  - onOpen
2021-11-01 14:43:27,278 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:28,283 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:29,285 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:30,292 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:31,306 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:32,307 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:33,313 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:34,314 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:35,321 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:36,329 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:37,336 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:38,338 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:39,349 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:40,351 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:41,352 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:42,353 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:43,357 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:44,359 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:45,362 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:46,364 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:47,368 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:48,373 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:49,377 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:50,379 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:51,383 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:52,395 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:53,402 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:54,405 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:55,409 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:56,417 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:57,422 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:58,424 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:43:59,428 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:00,430 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:01,431 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:02,433 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:03,438 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:04,441 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:05,445 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:06,449 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:07,450 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:08,455 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:09,460 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:10,576 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:11,579 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:12,581 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:13,584 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:14,585 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:15,593 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:16,596 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:17,600 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:18,601 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:19,609 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:20,612 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:21,618 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:22,622 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:23,626 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:24,632 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:25,637 DEBUG com.tercen.flowjo.TercenWebSocketListener  - send ping
2021-11-01 14:44:26,415 DEBUG com.tercen.flowjo.TercenWebSocketListener  - onFailure
java.net.SocketException: Socket closed
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:425)
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:65)
    at java.base/sun.security.ssl.SSLSocketImpl.bytesInCompletePacket(SSLSocketImpl.java:918)
    at java.base/sun.security.ssl.AppInputStream.read(AppInputStream.java:144)
    at okio.InputStreamSource.read(JvmOkio.kt:90)
    at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
    at okio.RealBufferedSource.request(RealBufferedSource.kt:206)
    at okio.RealBufferedSource.require(RealBufferedSource.kt:199)
    at okio.RealBufferedSource.readByte(RealBufferedSource.kt:209)
    at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.kt:119)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:102)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
    at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.base/java.lang.Thread.run(Thread.java:844)
2021-11-01 14:44:26,480 DEBUG com.tercen.flowjo.UploadProgressTask  - get schema
access-control-allow-headers x-ijt, content-type, authorization, etag, If-Match
access-control-allow-methods GET, OPTIONS, POST, PUT, DELETE
access-control-allow-origin *
access-control-max-age 3600
cache-control must-revalidate
content-type application/tson
date Mon, 01 Nov 2021 13:44:26 GMT
server nginx/1.19.2
strict-transport-security max-age=15724800; includeSubDomains
ServiceError(statusCode : 400 -- error : Schema.couchdb.id.missing -- reason : id is required)
    at com.tercen.service.HttpClientService.onResponseError(HttpClientService.java:39)
    at com.tercen.service.HttpClientService.get(HttpClientService.java:74)
    at com.tercen.flowjo.UploadProgressTask.uploadFile(UploadProgressTask.java:151)
    at com.tercen.flowjo.Utils.uploadCsvFile(Utils.java:60)
    at com.tercen.flowjo.Tercen.invokeAlgorithm(Tercen.java:211)
    at com.treestar.flowjo.engine.ExternalEngine.unsynchedSubmitQuery(Unknown Source)
    at com.treestar.flowjo.engine.ExternalEngine.submitQuery(Unknown Source)
    at com.treestar.flowjo.engine.CachingEngine.unsynchedSubmitQuery(Unknown Source)
    at com.treestar.flowjo.engine.ConcurrentCachingEngine.submitQuery(Unknown Source)
    at com.treestar.flowjo.engine.Query.executeQuery(Unknown Source)
    at com.treestar.flowjo.engine.Query.executeQuery(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode.executeQueryAndProcess(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode.access$100(Unknown Source)
    at com.treestar.flowjo.core.nodes.templating.ExternalPopNode$1.run(Unknown Source)
    at com.treestar.lib.util.STaskExecutor$2.run(Unknown Source)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.base/java.lang.Thread.run(Thread.java:844)
ginberg commented 3 years ago

should be fixed in https://github.com/tercen/flowjo_plugin/commit/b09a4ad1e7221a93c9efe3a8718f5538f8585f14