search

getodk / briefcase

ODK Briefcase is a Java application for fetching and pushing forms and their contents. It helps make billions of data points from ODK portable. Contribute and make the world a better place! ✨💼✨
https://docs.getodk.org/briefcase-intro
Other
60 stars 156 forks source link

Socket closed when pushing 3k+ submissions to Central #877

Open lognaturel opened 3 years ago

lognaturel commented 3 years ago

Software versions

Briefcase v1.17.4, macOS 10.15.4, Java 11

Problem description

2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO  o.a.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://test.central.getodk.org:443: Socket closed
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO  o.a.http.impl.execchain.RetryExec - Retrying request to {s}->https://test.central.getodk.org:443
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-1] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.net.SocketException: Socket closed
    at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:149)
    at org.opendatakit.briefcase.reused.http.CommonsHttp.execute(CommonsHttp.java:96)
    at org.opendatakit.briefcase.push.central.PushToCentral.pushSubmission(PushToCentral.java:270)
    at org.opendatakit.briefcase.push.central.PushToCentral.lambda$null$13(PushToCentral.java:160)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.net.SocketException: Socket closed
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
Caused by: java.net.SocketException: Socket closed

    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
    at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
    at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:142)
    ... 16 common frames omitted

Steps to reproduce the problem

Try to push 3k+ submissions to a Central server. This has happened to me several times and I have to close Briefcase and restart the transfer.

Expected behavior

No crash.

Other information

I haven't started doing any exploration. Maybe it's a concurrency issue? Or could the server be taking too long to respond at some point? It may very well just be intermittent and have nothing to do with the number of submissions sent.

It seems that with the same data set it failed twice at 4113 submissions sent. 4105 with the same form definition but different set of submissions. Sent several batches of ~500 forms and then an attempt to push 3000 failed at 1114. It's starting to feel like a resource management issue on the Briefcase end of things.

lognaturel commented 3 years ago

@getodk/testers here's a form and submissions that I was getting the failure for. It would be great to try on Windows and get a sense of whether it might be Mac-only. I'm on macOS 10.15.4, Java 11.

kkrawczyk123 commented 3 years ago

I was able to push to Central without any problems, none crash was visible on Windows 10. Screenshot from 2020-11-03 17-47-27

lognaturel commented 3 years ago

Confirming that on master as of right now (dd75f13) I'm still getting the same crash. Latest attempt was at 4402 submissions.