search

getodk / briefcase

ODK Briefcase is a Java application for fetching and pushing forms and their contents. It helps make billions of data points from ODK portable. Contribute and make the world a better place! ✨💼✨
https://docs.getodk.org/briefcase-intro
Other
60 stars 154 forks source link

Pushing large data set results in Too many open files error #848

Closed yanokwa closed 3 years ago

yanokwa commented 4 years ago

Software versions

Briefcase v1.17.1, macOS 10.15.2

Problem description

Pushing a large data set to Aggregate results in Too many open files error.

I believe that these errors cause the push to stop entirely.

I cannot reproduce this on Ubuntu server, which suggests to me that this is a macOS-specific issue. Documentation would be a fine fix for this, but if we can work around it in code, that'd be better.

Other information

2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-15] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files
    at org.opendatakit.briefcase.reused.UncheckedFiles.newInputStream(UncheckedFiles.java:300)
    at org.opendatakit.briefcase.reused.transfer.AggregateServer.getPushSubmissionRequest(AggregateServer.java:185)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.pushSubmissionAndAttachments(PushToAggregate.java:185)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.pushSubmissionAndAttachments(PushToAggregate.java:175)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:98)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
    at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
    at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
    at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
    at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
    at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
    at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
    at java.base/java.nio.file.Files.newInputStream(Files.java:155)
    at org.opendatakit.briefcase.reused.UncheckedFiles.newInputStream(UncheckedFiles.java:298)
    ... 29 common frames omitted
2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-5] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/uuidaa63b5ff-c5b0-4137-9ed4-aebd28b89f1b: Too many open files
    at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:278)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.getSubmissionAttachments(PushToAggregate.java:114)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:96)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
    at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
    at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
    at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/uuidaa63b5ff-c5b0-4137-9ed4-aebd28b89f1b: Too many open files
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:408)
    at java.base/java.nio.file.Files.newDirectoryStream(Files.java:471)
    at java.base/java.nio.file.Files.list(Files.java:3694)
    at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:276)
    ... 27 common frames omitted
Error pushing a form: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files (see the logs for more info)2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-3] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/BC2X5M0HFH25MEHPM28IG8HAH: Too many open files
    at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:278)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.getSubmissionAttachments(PushToAggregate.java:114)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:96)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
    at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
    at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
    at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
    at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/BC2X5M0HFH25MEHPM28IG8HAH: Too many open files
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:408)
    at java.base/java.nio.file.Files.newDirectoryStream(Files.java:471)
    at java.base/java.nio.file.Files.list(Files.java:3694)
    at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:276)
    ... 27 common frames omitted
poketim commented 4 years ago

Depending on the OS, the file systems will be different in the number of opened files allowed. https://linux.101hacks.com/unix/java-too-many-open-files/

In AggregateServer.java, on method getPushSubmissionRequest(), it's possible new InputStream() is called enough where it surpasses the allowed number of opened files.

To reproduce this problem, is the number of attachments large in the POST request?

One option I can think of is to perform a batch upload with a limit on file/attachments. So if you need 100 attachments uploaded, you can chunk them out and upload 10 at a time or however many at a time until you get through all 100, while being under the # of opened files limit.

Another option would be not to use new InputStream() and instead use classes that support multi-part file upload. https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileUpload.html That way, the files don't have to be opened and can still be submitted to the server. Although, this option may be more difficult because it will require more refactoring in other areas.

lognaturel commented 4 years ago

@timkoon, your explanation sounds just right. I talked to @yanokwa about the data he used to reproduce and he doesn't think that there were many attachments, just many submissions.

The first thing to check, I think, would be that streams are closed immediately upon successful submission.

poketim commented 4 years ago

@opendatakit-bot claim