Zips created don't extract in OS X

[steveandroulakis]

Observed in Mac OS X: 10.6, 10.7, 10.8

This zip file (1) was created clicking download all (zip) in the latest Mytardis:

https://dl.dropbox.com/u/172498/zips/mytardis-experiment36-complete.zip

This zip file (2) was created using the OS X gui:

https://dl.dropbox.com/u/172498/zips/36-manual-osx.zip

Zip file (1) fails to extract in OS X with an 'operation not permitted' error. However, using OS X's built in command-line 'unzip' application (identical to the Linux binary), works fine.

Zip file (2) unzips fine in all scenarios tested.

OS X usually doesn't have a problem unzipping zip files downloaded or otherwise originated elsewhere.

What's different about the new Zip process that means OS X can't read the file? I'm aware that Stephen Crawley mightn't have access to a mac to test the result of this. Nonetheless, I'd like his comment, as he coded the feature :)

original LH ticket

This ticket has 0 attachment(s).

[steveandroulakis]

Zips created don’t extract in OS X

The tardis download ZIP file creator currently uses a feature of ZIP files that allows you to put a zero file size and CRC into the "file header", and put the real file size and CRC into the "data descriptor" that follows the file data. This is controlled by bit 3 of the "general purpose bit flag" in the file header. See http://www.pkware.com/documents/casestudies/APPNOTE.TXT

You can see the code differences by comparing the ’write’ method of ’StreamableZipFile’ (in "tardis_portal/downloads.py") with the ’write’ method of the standard python ’ZipFile’ class.

If you recall, this change was made to fix a race condition in the previous ZIP file streamer that lead to ZIP file entries with bad headers. The alternative I proposed originally was to not stream the ZIP files at all; i.e. to generate them completely before allowing the download to start. That certainly would have been simpler ...

by Stephen Crawley

[steveandroulakis]

Zips created don’t extract in OS X

Other people have reported similar problems with the OS X Zip viewer and streaming mode Zip files; e.g. http://stackoverflow.com/questions/6507984/streaming-zip-file-wont-open-in-osx.

So what do you want me to do about it Steve? Strip out the clever streaming stuff and do this the simple way? And wear the slower download times?

by Stephen Crawley

[steveandroulakis]

Zips created don’t extract in OS X

Hi Stephen,

Actually, right now I’m leaning towards removing the zip option for OS X based user agents. OS X can extract tar files fine by default..

I’m not sure if messing with user agents for this would be the End Of The World but I feel if I was presented a tar only option as an OS X user, I wouldn’t care. A double click extract is better than a failing zip and zip works great everywhere else thanks to your work..

by Steve Androulakis

[steveandroulakis]

Zips created don’t extract in OS X

I was thinking about this ...

It would be possible to stream ZIP files without using the bit-3 feature that Mac software can’t cope with. But it involves reading and buffering each component file so that you can determine its size, compressed size and CRC >>before<< you write the header. That’s either going to require an extra FS round trip, or buffering a potentially large datafile in memory.

And what we have now is better than what we had before ... which was ZIP files that wouldn’t read anywhere :-)

by Stephen Crawley