Error: end of central directory record signature not found

[matthew-white]

There have been intermittent test failures with the error Error: end of central directory record signature not found. @ktuite and I have seen the error locally, and we also sometimes see the error on CircleCI. The error has been mentioned on the forum: https://forum.getodk.org/t/error-end-of-central-directory-record-signature-not-found/36182

We've seen two versions of the error:

Version 1:

     Error: end of central directory record signature not found
      at node_modules/yauzl/index.js:183:14
      at Immediate.<anonymous> (node_modules/yauzl/index.js:615:38)
      at processImmediate (internal/timers.js:464:21)

Version 2:

     Error: end of central directory record signature not found
      at node_modules/yauzl/index.js:183:14
      at node_modules/yauzl/index.js:622:5
      at node_modules/fd-slicer/index.js:32:7
      at FSReqCallback.wrapper [as oncomplete] (fs.js:561:5)

Version 1 of the error can be seen in this CircleCI build. I ran make test in CircleCI 200 times, and the error came up 6 times (all version 1). In CircleCI, we have seen the error in the following tests:

List

``` 1) zipPart streamer should close the archive only after parts are finalized: 1) managed encryption end-to-end should decrypt attached files successfully: 2) managed encryption end-to-end should handle mixed [plaintext/encrypted] attachments (not decrypting): 1) .zip attachments streaming should stream the contents to files at the appropriate paths: 1) api: /forms/:id/submissions [draft] .csv.zip should not carry over drafts when a draft is replaced: ```

@ktuite mentioned:

I occasionally see this Error: end of central directory record signature not found error on submission zip tests. Not all the time, not always the same test. But the tests all involve yauzl.

@ktuite has seen version 1 of the error in the following test:

  1) api: /forms/:id/submissions
       [draft] .csv.zip
         should return draft submissions:

I've seen version 2 of the error locally in the following tests:

List

``` 2) managed encryption end-to-end should decrypt over cookie auth with passphrases provided via url-encoded POST body: 1) managed encryption end-to-end should strip .enc suffix from decrypted attachments: 1) managed encryption end-to-end should decrypt attached files successfully: 2) managed encryption end-to-end should decrypt to CSV successfully: 1) zipPart streamer should close the archive only after parts are finalized: 2) .csv.zip briefcase output @slow should output all rows: 1) .csv.zip briefcase output @slow should attach attachments information if present: 1) api: /forms/:id/submissions .csv.zip GET versioning should return original instanceId and latest attached audit log when instanceId deprecated with new audit log: 1) api: /forms/:id/submissions .csv.zip GET should return consolidated client audit log filtered by user: 1) api: /forms/:id/submissions .csv.zip GET should return consolidated client audit log filtered by review state: 1) api: /forms/:id/submissions .csv.zip GET should not include data from other forms: 1) api: /forms/:id/submissions [draft] .csv.zip should not carry draft submissions forward to the published version upon publish: ```

The error mentioned on the forum is also version 2 of the error.

[matthew-white]

I think there's some issue with unzipping (either with the package we use or with how we use it). It's rare that we have to unzip in production. (Maybe backup restore is the only case?) But it's common for us to unzip in testing.

[matthew-white]

I ran make test in CircleCI another 100 times after merging #591. That PR upgraded Node and also bumped yauzl, a dependency seen in the stack traces above. However, I'm still seeing this error, so it doesn't seem to be specific to Node 14.

[alxndrsn]

Seen in CircleCI at https://app.circleci.com/pipelines/github/getodk/central-backend/1131/workflows/f9ff55b2-5136-4834-814d-edb7a1f7ab10/jobs/1912:

  1) managed encryption
       end-to-end
         should handle mixed[managedA/managedB] formdata (decrypting):
     Error: end of central directory record signature not found
      at /home/circleci/repo/node_modules/yauzl/index.js:187:14
      at Immediate.<anonymous> (node_modules/yauzl/index.js:624:38)
      at processImmediate (node:internal/timers:466:21)

make: *** [Makefile:35: test-full] Error 1

[alxndrsn]

New variation seen at https://app.circleci.com/pipelines/github/alxndrsn/odk-central-backend/182/workflows/25663dd0-3856-4c4a-9ebc-b132a26a39dc/jobs/189

  1 failing

  1) managed encryption
       end-to-end
         should handle mixed[plaintext/encrypted] formdata (not decrypting):
     Error: end of central directory record signature not found
      at /home/circleci/repo/node_modules/yauzl/index.js:187:14
      at Immediate.<anonymous> (node_modules/yauzl/index.js:624:38)
      at processImmediate (node:internal/timers:466:21)

make: *** [Makefile:39: test-full] Error 1

Exited with code exit status 2

[alxndrsn]

"end of central directory record signature not found" suggests that pZipStreamToFiles() (or maybe zipStreamToFiles()) is being provided a corrupt/invalid zipfile.

This suggests there was an error in the backend when constructing the zipfile.

This should be recreatable by throwing in e.g. Writable.write() in streamAttachments() in lib/data/attachments.js. There does not currently appear to be any logging of these errors.

Streaming responses are confusing in part because the response headers & status are written before the content of the response has been fetched from the DB / generated.