search

strawlab / strand-braid

Live, low-latency 2D and 3D tracking from single or multiple high-speed cameras
https://strawlab.org/braid/
Other
50 stars 8 forks source link

Corrupt braidz (zip) file #5

Closed elhananby closed 3 years ago

elhananby commented 3 years ago

Hi Andrew, Recently see more and more often an issue with the braidz file saved from the system.

When opening it with zipfile in python, I get this error: BadZipFile: Truncated file header

If I change the extension to .zip, I can view the contents using the archive viewer in Ubuntu, and I can see all the files are in there, but if I try to extract it using unzip -t I get this error: warning [20210618_132644.zip]: 12884901888 extra bytes at beginning or within zipfile (attempting to process anyway) file #1: bad zipfile offset (local header sig): 12884902024 (attempting to re-compensate) testing: README.md OK error: not enough memory for bomb detection At least one error was detected in 20210618_132644.zip.

The output from braidz did not show any issues in creating the file: Jun 20 21:48:31.524 INFO flydra2_mainbrain: stopping saving Jun 20 21:48:32.158 INFO flydra2: creating zip file /media/benyishay_la/Data/Experiments/20210618_132644.braidz Jun 20 21:48:47.696 INFO flydra2: done creating zip file, removing /media/benyishay_la/Data/Experiments/20210618_132644.braid

I was able to "fix" the archive using the command from this post: https://unix.stackexchange.com/a/115831, but then the file size got reduced from 15GB (yeah, it's a very long recording) to about 7GB. I've attached the output from this command, maybe it contains some helpful information? zip_FFv_fix.txt

Thanks, Elhanan

astraw commented 3 years ago

We haven't tested much with files beyond about 3 GB so my first thought is that there may be some bug associated with giant files that we haven't hit here. Do you see these errors in 100% of cases when the files exceed a certain size? Or do you sometimes have e.g. 15GB files with no problems?

astraw commented 3 years ago

Prompted by your report, I went digging into large file support within the zip library used by braid. It seems large file support (files greater than 4GB) was added just a few weeks ago and I went ahead and added support for it in d60b0dac911d653dffb0544a732e3832bb2dbde4. While doing this, I also noticed another bug in the code and fixed that as well.

I will make a new release soon which incorporates this fix.

I am assuming this will fix your issue, but if it does not, please re-open this.

astraw commented 3 years ago

I released 0.9.1 which should fix the issue.