Higher zip file corruption rates with aes256 option enabled

[abought]

Summary

TIS has received a small but steady stream of bug reports involving zip file corruption, most commonly when using the AES256 encryption option.

EDIT: There are some reports of corruption with regular zip files as well (including for small files). It is not clear whether zip compression is directly the cause, or whether other factors might be in play. (filesystem issues during the merge step, etc)

What we know

Because this happens with user provided sensitive data, we don't own a reproducible test case that can be shared- this is a longstanding heisenbug with occurrences going back at least 1-2 yr (limit of my time on the project). This also makes it difficult to assess actual rate, though we received multiple issue reports quite recently.

• Most likely to occur in the largest files (first few chromosomes), but not exclusively so (also reported in chrX)
  • Does not occur at any specifically significant filesize boundary. (for example, a ~77GB file might be corrupt and a 70GB file might be ok)
• Not a problem with passwords (other zip files from the same run extract ok, even with same compression type)
• Not a problem with file corruption; downloaded files had same hash as server-provided values
• Some users report seeing the problem happen again for the same files if they re-impute with same compression settings (maybe not a totally random aspect)
• The problem typically goes away if they re-impute without AES256 encryption (using just the weak native zip option)

Initial workup

• No specific indications of known corrupt AES256 archives in zip4j bug reports based on our current version. However, there have been reports of other corrupt archives in the past.

Open questions

• Are there known issues / has this been seen in MIS?
• Are there potential mitigations we can apply to reduce the incidence?

[seppinho]

Thanks for the bug report. Unfortunately we don't have a fix for that right now (probably related to the zip4j package). We already replaced the zip4j package in the upcoming pipeline version with the commandline version of 7z (using the -mem=AES256 option).

Fyi: @cfuchsberger mentioned in the past, changing the header information often helped to bypass this problem.