search

getodk / briefcase

ODK Briefcase is a Java application for fetching and pushing forms and their contents. It helps make billions of data points from ODK portable. Contribute and make the world a better place! ✨💼✨
https://docs.getodk.org/briefcase-intro
Other
60 stars 154 forks source link

CLI can't find metadata.json file unless previously created by GUI #824

Closed chrissyhroberts closed 4 years ago

chrissyhroberts commented 4 years ago

If GUI has not previously created the ODK Briefcase Storage Directory and CLI is used to pull, briefcase creates the new folder, giving message

The directory ./foo/ODK Briefcase Storage doesn't exist. Creating it

But later sends error such as

Error pulling a form: java.nio.file.NoSuchFileException: ./foo/ODK Briefcase Storage/./foo/ODK Briefcase Storage/forms/foo/metadata.json (see the logs for more info)

In logs this kind of thing

2019-10-15 10:42:55,816 [ForkJoinPool-1-worker-9] ERROR o.o.b.o.PullFormFromAggregate - Error pulling a form java.io.UncheckedIOException: java.nio.file.NoSuchFileException: ./foo/ODK Briefcase Storage/./foo/ODK Briefcase Storage/forms/formid/metadata.json at org.opendatakit.briefcase.reused.UncheckedFiles.write(UncheckedFiles.java:80) at org.opendatakit.briefcase.reused.UncheckedFiles.write(UncheckedFiles.java:73) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.serialize(FileSystemFormMetadataAdapter.java:123) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.persist(FileSystemFormMetadataAdapter.java:96) at org.opendatakit.briefcase.model.form.FormMetadataCommands.lambda$updateAsPulled$1(FormMetadataCommands.java:19) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.execute(FileSystemFormMetadataAdapter.java:43) at org.opendatakit.briefcase.pull.aggregate.PullFromAggregate.lambda$pull$11(PullFromAggregate.java:155) at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134) at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65) at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1603) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) Caused by: java.nio.file.NoSuchFileException: ./foo/ODK Briefcase Storage/./foo/ODK Briefcase Storage/forms/formid/metadata.json at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.base/java.nio.file.Files.newOutputStream(Files.java:218) at java.base/java.nio.file.Files.write(Files.java:3355) at org.opendatakit.briefcase.reused.UncheckedFiles.write(UncheckedFiles.java:78) ... 12 common frames omitted 2019-10-15 10:44:32,344 [main] ERROR o.o.briefcase.ui.MainBriefcaseWindow - Failed to launch GUI java.io.UncheckedIOException: java.nio.file.NoSuchFileException: ./foo/ODK Briefcase Storage/./foo/ODK Briefcase Storage/forms/formid/metadata.json at org.opendatakit.briefcase.reused.UncheckedFiles.write(UncheckedFiles.java:80) at org.opendatakit.briefcase.reused.UncheckedFiles.write(UncheckedFiles.java:73) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.serialize(FileSystemFormMetadataAdapter.java:123) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.persist(FileSystemFormMetadataAdapter.java:96) at java.base/java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:441) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1492)

Looks like CLI is trying to update the .json file used for tracking cursor positions, but doesn't know what to do if the file is not present.

ggalmazor commented 4 years ago

Thanks for reporting this, @chrissyhroberts!

~Briefcase should create the ./foo/ODK Briefcase Storage/forms subdirectory instead of just the ./foo/ODK Briefcase Storage directory. I'm already working on a fix~ (sorry, wrong issue. That's another one of the upcoming v2.0)

ggalmazor commented 4 years ago

Hi again!

I've tried to reproduce the issue without success. I'm guessing you were trying to pull all the forms present in the server with a CLI command like this one: java -jar briefcase.jar -plla -p aggregate -u administrator -sd /tmp/cocotero -U https://sandbox.aggregate.opendatakit.org

I'm using the sandbox right now, which has several forms. I'm running Briefcase from a clean state (clean prefs, non-existing storage directory).

Is there something different than what you're doing to get the error?

chrissyhroberts commented 4 years ago

Not sure what to suggest for reproduction, but this post on forum seems to have the same issue.

https://forum.opendatakit.org/t/problems-with-exporting-to-csv-via-cli/22786

One think I have started to wonder is if some of these incompatibilities are due to using new versions of Briefcase with older versions of Aggregate. i.e. is there something different in the way that Aggregate behaves in newer versions that could explain this. The weird effects that many of our users have seen where exports don't complete if you use new briefcase + old aggregate are definitely due to version incompatibilities between briefcase & aggregate (even at the level of exporting)

ggalmazor commented 4 years ago

Unfortunately, it's more probable that we(I) have introduced unexpected changes that could justify these issues. Thanks for the link to the forum post!

ggalmazor commented 4 years ago

FWIW, I think the root cause for the error we're getting could be related to highly concurrent access to disk.

Java is complaining about a directory that should exist (the form's directory in the SD) but it's not there when it's ready to write the metadata.json file.

A quick fix would be to ensure the directory exists before that, but I'd be much happier if I could isolate a scenario that reproduces the issue to be sure I'm fixing the issue once-and-for-all and put a regression test in place.

ggalmazor commented 4 years ago

It's also strange that the sd path appears twice in the error, which could tells us that we're building paths incorrectly at some point.

Also, something noteworthy is that the sd you're defining is a relative path. This could be key.

@chrissyhroberts, could you maybe share the contents of one of the metadata.json files of a form that's failing?

ggalmazor commented 4 years ago

OK, the relative path as sd does the trick. Now I can reproduce :)

chrissyhroberts commented 4 years ago

Ah so if path were explicit it would pass. That's interesting.

I wonder if something similar could explain the issue of exports failing, which started happening in the same version release.

ggalmazor commented 4 years ago

I wonder if something similar could explain the issue of exports failing, which started happening in the same version release.

I'll check that too. Thanks!

ggalmazor commented 4 years ago

Would you try this JAR to see if the error persist, @chrissyhroberts?

briefcase_i824.zip

ggalmazor commented 4 years ago

Regarding the export issue:

chrissyhroberts commented 4 years ago

With 1.17.0

Error pulling a form: java.nio.file.NoSuchFileException: ./MSC_MAPPING_EPI2/ODK Briefcase Storage/./MSC_MAPPING_EPI2/ODK Briefcase Storage/forms/MSCEPI_MAPPING_ODK/metadata.json (see the logs for more info)

With this beta everything seems to work with regards pulling

chrissyhroberts commented 4 years ago

Export still stalls, but this is data downloaded from an old version of Aggregate (v1.4.15 Production)

Exporting form MSCEPI_MAPPING_ODK (MSC_MAPPING_EPI) to: ./MSC_MAPPING_EPI3
2019-10-21 16:19:23,786 [ForkJoinPool-1-worker-9] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
org.opendatakit.briefcase.model.CryptoException: org.opendatakit.briefcase.model.CryptoException: Can't decrypt file
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
    at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
    at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:575)
    at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:111)
    at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:56)
    at org.opendatakit.briefcase.operations.Export.lambda$export$7(Export.java:156)
    at org.opendatakit.briefcase.reused.job.Job.lambda$run$0(Job.java:54)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$7(Job.java:123)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$7(Job.java:122)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
    at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1603)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: org.opendatakit.briefcase.model.CryptoException: Can't decrypt file
    at org.opendatakit.briefcase.export.SubmissionParser.decryptFile(SubmissionParser.java:213)
    at org.opendatakit.briefcase.export.SubmissionParser.decrypt(SubmissionParser.java:192)
    at org.opendatakit.briefcase.export.SubmissionParser.lambda$parseSubmission$5(SubmissionParser.java:149)
    at java.base/java.util.Optional.flatMap(Optional.java:283)
    at org.opendatakit.briefcase.export.SubmissionParser.parseSubmission(SubmissionParser.java:129)
    at org.opendatakit.briefcase.export.ExportTools.lambda$getValidSubmissions$0(ExportTools.java:30)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1492)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    ... 3 common frames omitted
Caused by: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted
    at java.base/javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:121)
    at java.base/javax.crypto.CipherInputStream.read(CipherInputStream.java:235)
    at java.base/javax.crypto.CipherInputStream.read(CipherInputStream.java:212)
    at org.opendatakit.briefcase.export.SubmissionParser.decryptFile(SubmissionParser.java:208)
    ... 13 common frames omitted
Caused by: javax.crypto.BadPaddingException: pad block corrupted
    at org.bouncycastle.jce.provider.JCEBlockCipher.engineDoFinal(Unknown Source)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2074)
    at java.base/javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:118)
    ... 16 common frames omitted
Error exporting a form: org.opendatakit.briefcase.model.CryptoException: Can't decrypt file (see the logs for more info)
2019-10-21 16:19:23,794 [ForkJoinPool-1-worker-9] ERROR o.o.briefcase.operations.Export - Error exporting a form
org.opendatakit.briefcase.model.CryptoException: org.opendatakit.briefcase.model.CryptoException: Can't decrypt file
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
    at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
    at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:575)
    at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:111)
    at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:56)
    at org.opendatakit.briefcase.operations.Export.lambda$export$7(Export.java:156)
    at org.opendatakit.briefcase.reused.job.Job.lambda$run$0(Job.java:54)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$7(Job.java:123)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$7(Job.java:122)
    at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
    at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1603)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: org.opendatakit.briefcase.model.CryptoException: Can't decrypt file
    at org.opendatakit.briefcase.export.SubmissionParser.decryptFile(SubmissionParser.java:213)
    at org.opendatakit.briefcase.export.SubmissionParser.decrypt(SubmissionParser.java:192)
    at org.opendatakit.briefcase.export.SubmissionParser.lambda$parseSubmission$5(SubmissionParser.java:149)
    at java.base/java.util.Optional.flatMap(Optional.java:283)
    at org.opendatakit.briefcase.export.SubmissionParser.parseSubmission(SubmissionParser.java:129)
    at org.opendatakit.briefcase.export.ExportTools.lambda$getValidSubmissions$0(ExportTools.java:30)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1492)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    ... 3 common frames omitted
Caused by: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted
    at java.base/javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:121)
    at java.base/javax.crypto.CipherInputStream.read(CipherInputStream.java:235)
    at java.base/javax.crypto.CipherInputStream.read(CipherInputStream.java:212)
    at org.opendatakit.briefcase.export.SubmissionParser.decryptFile(SubmissionParser.java:208)
    ... 13 common frames omitted
Caused by: javax.crypto.BadPaddingException: pad block corrupted
    at org.bouncycastle.jce.provider.JCEBlockCipher.engineDoFinal(Unknown Source)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2074)
    at java.base/javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:118)
    ... 16 common frames omitted

All operations completed
yanokwa commented 4 years ago

Would you try this JAR to see if the error persist, @chrissyhroberts?

briefcase_i824.zip

I can confirm that this version worked for me, @ggalmazor.

chrissyhroberts commented 4 years ago

Sorry I thought I had already fed back on this one! Too many things going on at the moment. Yes, as far as I can tell this does work. Thanks!