Problems with Zip Files after upgrade to 7.0.0

maddox-DX commented 10 years ago

Steps to reproduce

Login und go to files
select a couple of files download the generated zip file
try to decompress with Total Commander
Expected behaviour

Files should be extracted

Actual behaviour

Total Commander gives CRC warnings

Server configuration

Operating system: debian 7

Web server: apache 2.2 Database: postgres 9.1 PHP version: 5.4 ownCloud version: (see ownCloud admin page) 7.0.0 Updated from an older ownCloud or fresh install: From 6.0.4

I got an hint from the Total Commander forum: http://ghisler.ch/board/viewtopic.php?t=40766 (in german)

The linked forum post says that the generated zip has two issues:

the content is not compressed
the file entries in the zip file don't provide valid crc data (and thats why total commander complains).

ghost commented 10 years ago

OSX also says that the downloaded zip file is not a valid zip file.

DeepDiver1975 commented 10 years ago

@McNetic can you please have a look at this? This is related to the zip streamer component

MTRichards commented 10 years ago

Moved to 7.0.2 for investigation.

McNetic commented 10 years ago

Ok. I had a look at the issue. First of all, this is two separate problems, one with TotalCommander, one with OSX finder.

Regarding the TotalCommander issue: 1) The content is not compressed. This is totally compliant with the zip standard, and is a known issue already discussed elsewhere (as php does provide no known solution to do chunk-wize deflate compression). 2) Totalcommander 8.51a complains about crc errors. The linked forum post suggests that crc is missing from the zip file. AFAICS: That is not true. The crc is in place and correct. What indeed is "missing" is the compressed/uncompressed file size information in the header (in fact, it is set to -1, as the zip standard demands for stream compression). There was an issue with Totalcommander in past versions (see https://github.com/McNetic/PHPZipStreamer/issues/8). Back then, I tested with v8.50 and had not problems. I don't have v8.50 any more, and in more recent versions, the zip handling was changed according to the changelog, so I suspect there went something wrong. I'll try to sort this out with the Totalcommander author.

Regarding the OSX finder issue: I'm afraid OSX finder is not able to handle this kind of (standards-compliant) zip files correctly. This is a known bug in OSX and Zip64 extension - search google for "osx zip cpgz"). Other software on OSX works fine (see https://github.com/McNetic/PHPZipStreamer/issues/4). I'd be very pleased if someone had an idea how to make OSX finder happy.

gem76 commented 10 years ago

Hi, i started the thread on the totalcommander-board, i would like to cross-refrence this one to bring it to the attention of the author of tc.

The ZIP-Files produced by OC6 didn't cause this error in TC - and if i remember correct, they were compressed. As i've mentioned in the tc-thread, other tools handle the archives just well.

McNetic commented 10 years ago

The implementation changed. The reasoning behind this is explained in detail here: https://github.com/owncloud/core/pull/3439, some more background is also available here: https://github.com/owncloud/core/pull/6893, and the actual merge was done here: https://github.com/owncloud/core/pull/7328.

ghost commented 10 years ago

I have the same issue on FreeBSD (as noted in the duplicate ticket). Is there any troubleshooting I need to do for this?

Is there a way to stop the zip creation and maybe just download them all separately.

ldaverio commented 10 years ago

In my opinion, it's not a Finder-related issue, the problem exists with the command line too:

$ which unzip /usr/bin/unzip

$ unzip download.zip Archive: download.zip warning [download.zip]: zipfile claims to be last disk of a multi-part archive; attempting to process anyway, assuming all parts have been concatenated together in order. Expect "errors" and warnings...true multi-part support doesn't exist yet (coming soon). error [download.zip]: missing 8469400417 bytes in zipfile (attempting to process anyway) error [download.zip]: attempt to seek before beginning of zipfile (please check that you have transferred or created the zipfile in the appropriate BINARY mode and that you have compiled UnZip properly)

$ unzip -v UnZip 5.52 of 28 February 2005, by Info-ZIP. Maintained by C. Spieler. Send bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ; see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68) for Unix on Aug 24 2013.

UnZip special compilation options: COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported) SET_DIR_ATTRIB TIMESTAMP USE_EF_UT_TIME USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported) USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported) VMS_TEXT_CONV [decryption, version 2.9 of 05 May 2000]

UnZip and ZipInfo environment options: UNZIP: [none] UNZIPOPT: [none] ZIPINFO: [none] ZIPINFOOPT: [none]

DeepDiver1975 commented 10 years ago

@McNetic ^

McNetic commented 10 years ago

Afaics, your unzip version is too old. Obviously, Apple is packaging 9 year old software to handle a file format whose spec is a moving target.

nico@haven:~/programming/ZipStreamer$ unzip -v
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.7.2 for Unix (Linux ELF) on Nov 28 2012.

UnZip special compilation options:
        ACORN_FTYPE_NFS
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        SET_DIR_ATTRIB
        SYMLINKS (symbolic links supported, if RTL and file system permit)
        TIMESTAMP
        UNIXBACKUP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        UNICODE_SUPPORT [wide-chars, char coding: UTF-8] (handle UTF-8 paths)
        LARGE_FILE_SUPPORT (large files over 2 GiB supported)
        ZIP64_SUPPORT (archives using Zip64 for large files supported)
        USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)
        VMS_TEXT_CONV
        WILD_STOP_AT_DIR
        [decryption, version 2.11 of 05 Jan 2007]

UnZip and ZipInfo environment options:
           UNZIP:  [none]
        UNZIPOPT:  [none]
         ZIPINFO:  [none]
      ZIPINFOOPT:  [none]

There's obviously missing LARGE_FILE_SUPPORT (may not be needed in this special case), but in any case ZIP64_SUPPORT and also (unrelated to the issue, but also problematic) UNICODE_SUPPORT.

McNetic commented 10 years ago

Ok, I'd like to open a discussion on this one: Obviously, the problem can not be really solved. On the one hand, we need Zip64 support to be able to deliver zip files larger than 4GB. On the other hand, there's prominent software (especially MAC OSX) that can still not handle the Zip64 extension. The latter problem can be overcome by third party products (not all, and I don't remember which, but afair it's mentioned in one of the ZipStreamer issues).

I think staying with large file support is the way to go (as the default).
we could add an option to disable Zip64 support manually by the user (though I'm afraid most users won't understand when and why to enable this, and what the consequences may be).
I could make the Zip64 support in ZipStreamer optional, so Owncloud could summarize the file sizes and disable it when not required, so the problem only occurs for large files.

Which option shall we pick? Do you see other options?

@DeepDiver1975 could you notify whomever takes part in this decision?

BrentMartel commented 10 years ago

@McNetic - while I agree that it is frustrating that the default zip utility in OSX is out of date, I also think that we need to be sure that the burden is not on our OwnCloud end-users to have to worry about using alternate desktop zip utilities. My thought is that we would want the priority to be on end-user compatabilty over support for niche use cases. The need to support zip generation above 4GB is likely a rare one and can even be a server resource issue.

I think the best of all evils is sticking with the broadest zip client support possible to ensure maximum end-user compatability as a default.
Allow enabling of Zip64 (and thus 4GB+) as a config setting for specific installations that require it...i.e. corporate deployments where administrators know and can control their desktop environment.

Dropbox even limits the download as zip feature to 1GB and presents users with a nice message saying the contents are too large to download as zip: https://www.dropbox.com/en/help/49

ghost commented 10 years ago

The bad news This is form OS X Yosemite Beta (10.10):

$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.38) for Unix on Jun 28 2014.

UnZip special compilation options:
    COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
    SET_DIR_ATTRIB
    TIMESTAMP
    USE_EF_UT_TIME
    USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
    USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
    VMS_TEXT_CONV
    [decryption, version 2.9 of 05 May 2000]

UnZip and ZipInfo environment options:
           UNZIP:  [none]
        UNZIPOPT:  [none]
         ZIPINFO:  [none]
      ZIPINFOOPT:  [none]

And yes of course own cloud should try to support as many clients as possible. But I'm not sure if it's a good idea to switch between Zip64 support at the settings for the whole server.

Better would be something that let the user of the download decide like:

compatibility mode
advanced mode (support for > 4GB, ...) - not working everywhere

This would be especially fine for link-shared folders, where you can have much more users (and clients) then in a small personal setup.

Maybe this decision can be preselected by the user agent information.

LukasReschke commented 10 years ago

Ok, I'd like to open a discussion on this one: Obviously, the problem can not be really solved. On the one hand, we need Zip64 support to be able to deliver zip files larger than 4GB. On the other hand, there's prominent software (especially MAC OSX) that can still not handle the Zip64 extension. The latter problem can be overcome by third party products (not all, and I don't remember which, but afair it's mentioned in one of the ZipStreamer issues).

I think staying with large file support is the way to go (as the default). we could add an option to disable Zip64 support manually by the user (though I'm afraid most users won't understand when and why to enable this, and what the consequences may be).

I could make the Zip64 support in ZipStreamer optional, so Owncloud could summarize the file sizes and disable it when not required, so the problem only occurs for large files. Which option shall we pick? Do you see other options?

@karlitschek This is really an annoying issue to me and Mac users in general. I'd advise to go with the second approach (use zip64 if the overall filesize is >4GB, otherwise use the regular ZIP format that is understood by any zip version).

Users can live with the fact that large archives (>4GB) does not work with OS X. But broken downloads are just really annoying. Your decision please.

\cc @McNetic @DeepDiver1975

LukasReschke commented 10 years ago

also @MTRichards and @craigpg

karlitschek commented 10 years ago

Agreed. Can you provide a Patch?

MTRichards commented 10 years ago

@LukasReschke Agreed with you and @karlitschek . I like the second option as well. It provides most users with an easy path to making it all work with no intervention.

LukasReschke commented 10 years ago

@McNetic Could you make zip64 support in ZipStreamer optional? (e.g. enabled per default but can be disabled via the constructor) - I can then take care of the needed changes in ownCloud. Thanks a lot!

jens-maus commented 10 years ago

I was also hit by this issue under OSX and I also have to say that it is pretty much appreciated if it could be resolved. As for how to fix this and disabling zip64 support by default to support OSX clients I think it might also be a possibility to make zip64 support depending on the operating system a browser reports. So wouldn't it be simply possible to query the browser for the operating system being used and then disable zip64 support for downloads with OSX while other platforms will keep it per default? Would that be an option?

McNetic commented 10 years ago

I'll change the implementation of ZipStreamer to make Zip64 support optional, but I cannot promise a fixed date (I hope only a couple of days).
It should not at all be a performance issue on the server side. Afaics, the load of ZipStreamer does not change with bigger archives, as it works with fixed size chunks and sends them to the client immediately.
I like the option of detecting OSX and (perhaps) disabling zip64 support by default - or provide a small hint that explains the issue so power users can override and normal users understand what happens and why.
By the way, I have a patch quite ready that will enable (optional) deflate compression support. I'm still lacking unit tests for this; and I'm not really happy about the current unit tests (that compare output to some pre-generated zip files; as soon as zip options are changed, the output will change as well). I'd be more than happy if someone had a better idea how to implement the unit tests.

LukasReschke commented 10 years ago

I'll change the implementation of ZipStreamer to make Zip64 support optional, but I cannot promise a fixed date (I hope only a couple of days).

Great. Did you have time to look into this or is there something that we can do to help you? :)

I like the option of detecting OSX and (perhaps) disabling zip64 support by default - or provide a small hint that explains the issue so power users can override and normal users understand what happens and why.

From my understanding we should do the following checks and if one of them is positive we need to use ZIP64 and show a warning to users with an OS X UA:

Check if the files are >4GB
Check if there are more than 65’535 files

Or is there any downside if we use regular ZIP instead of ZIP64 as default?

By the way, I have a patch quite ready that will enable (optional) deflate compression support. I'm still lacking unit tests for this; and I'm not really happy about the current unit tests (that compare output to some pre-generated zip files; as soon as zip options are changed, the output will change as well). I'd be more than happy if someone had a better idea how to implement the unit tests.

Where can we see this changes? Which branch/PR?

@McNetic

guruz commented 10 years ago

@danimo You mentioned something about UTF8 characters in filenames?

danimo commented 10 years ago

@guruz Maybe I was wrong there. but didn't we also upgrade to have utf8-filenames? or is that independent from the 64 bit zip file format?

christianmatts commented 10 years ago

hi, I was setting up own cloud as a gui front end to send ftp links to clients. in os x 10.7.5 downloading multiple files (~256mb) resulted in a zip file that requires a 3rd party app to unzip - which is deal breaker. If we could set it to download multiple files without creating a zip, that would be helpful.

MTRichards commented 10 years ago

Did this get implemented? Not sure if this is resolved from the above.

LukasReschke commented 10 years ago

No. This still requires an upstream fix.

ghost commented 10 years ago

Is there any news on this issue?

trentster commented 10 years ago

As a semi permanent workaround for OSX users is to install and use the Free Unarchiver utility from the Mac app store. Its better for all uncompress work anyway. Been using it for years and it works like a dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353

This is something that should really be fixed from Apples side - they should not be bundling zip versions from 2005 that does not properly support large files or zip64 encoding.

christianmatts commented 10 years ago

(can i just reply to the web post via email?)

I have done this option for myself, but when sending links to files to clients, this doesnt active my goal of simplifying sending files hosted on our ftp site.

I was trying to avoid asking clients to download a specific software to download files - either a ftp client or an unarchiving tool.

Just about all of my clients are on a mac.

On 2014-09-11 10:12 am, trentster wrote:

As a semi permanent workaround for OSX users is to install and use the Free Unarchiver utility from the Mac app store. Its better for all uncompress work anyway. Been using it for years and it works like a dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]

This is something that should really be fixed from Apples side - the should not be bundling zip versions from 2005 that does not support large files or zip64

Reply to this email directly or view it on GitHub [2].

Links:

[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353 [2] https://github.com/owncloud/core/issues/10001#issuecomment-55269149

jens-maus commented 10 years ago

I think it has been discussed enough before that using an alternative unarchiver or waiting until Apple has fixed their zip support is no alternative. Thus, we need a short-term solution for the issue discussed here. So any progress on an improved Zip support in owncloud?

BrentMartel commented 10 years ago

Agreed, asking end-users to install anything special is never a solution.

I don't see any issue with cutting off support for outdated OS versions, but this is not the case.

Unfortunatly, Apple's laziness is something that needs to be worked around :-)

On Thu, Sep 11, 2014 at 10:23 AM, christianmatts notifications@github.com wrote:

(can i just reply to the web post via email?) I have done this option for myself, but when sending links to files to clients, this doesnt active my goal of simplifying sending files hosted on our ftp site. I was trying to avoid asking clients to download a specific software to download files - either a ftp client or an unarchiving tool. Just about all of my clients are on a mac. On 2014-09-11 10:12 am, trentster wrote:

As a semi permanent workaround for OSX users is to install and use the Free Unarchiver utility from the Mac app store. Its better for all uncompress work anyway. Been using it for years and it works like a dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]

This is something that should really be fixed from Apples side - the should not be bundling zip versions from 2005 that does not support large files or zip64

Reply to this email directly or view it on GitHub [2].

Links:

[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353

[2] https://github.com/owncloud/core/issues/10001#issuecomment-55269149

Reply to this email directly or view it on GitHub: https://github.com/owncloud/core/issues/10001#issuecomment-55270869

christianmatts commented 10 years ago

I'm on osx 10.7 … is that already outdate? If so myself/my office and most of my clients are outdated.

And I'm using the own cloud free product so i can't complain, but this isn't the first time support for products advertising os x support basically says the solution my problem is to use windows -- which isn't actually helpful.

On 2014-09-11 10:29 am, Cagex wrote:

Agreed, asking end-users to install anything special is never a solution.

I don't see any issue with cutting off support for outdated OS versions, but this is not the case.

Unfortunatly, Apple's laziness is something that needs to be worked around :-)

On Thu, Sep 11, 2014 at 10:23 AM, christianmatts notifications@github.com wrote:

(can i just reply to the web post via email?) I have done this option for myself, but when sending links to files to clients, this doesnt active my goal of simplifying sending files hosted on our ftp site. I was trying to avoid asking clients to download a specific software to download files - either a ftp client or an unarchiving tool. Just about all of my clients are on a mac. On 2014-09-11 10:12 am, trentster wrote:

As a semi permanent workaround for OSX users is to install and use the Free Unarchiver utility from the Mac app store. Its better for all uncompress work anyway. Been using it for years and it works like a dream. https://itunes.apple.com/au/app/the-unarchiver/id425424353 [1]

This is something that should really be fixed from Apples side - the should not be bundling zip versions from 2005 that does not support large files or zip64

Reply to this email directly or view it on GitHub [2].

Links:

[1] https://itunes.apple.com/au/app/the-unarchiver/id425424353 [2]

https://github.com/owncloud/core/issues/10001#issuecomment-55269149

Reply to this email directly or view it on GitHub: https://github.com/owncloud/core/issues/10001#issuecomment-55270869

Reply to this email directly or view it on GitHub [1].

Links:

[1] https://github.com/owncloud/core/issues/10001#issuecomment-55271676

McNetic commented 10 years ago

Just a small update: Life got in my way, but I'm on to the issue.

TomGem commented 10 years ago

Thanks! I can hardly wait to checkout your commit.

DeepDiver1975 commented 10 years ago

Just a small update: Life got in my way, but I'm on to the issue.

Thanks a lot @McNetic

lepetitmonde commented 10 years ago

hello, @McNetic did you find a solution ? is this for next version of the software as @MTRichards said ?

lepetitmonde commented 10 years ago

hello, nothing new about the zip problem on Mac ?

pcace commented 10 years ago

Really nothing new? Its really annoying since here on my server there are around 50 users using OSX....

LukasReschke commented 10 years ago

There is a work in progress version by @McNetic - I'm limiting this issue to collaborators since we all know that the issue is annoying but pinging every few days doesn't make it resolved faster. In fact, it notifies everybody in this issue which is a huge waste of time.

Thanks.

McNetic commented 10 years ago

The relevant issue in ZipStreamer is McNetic/PHPZipStreamer#12.

PVince81 commented 9 years ago

@DeepDiver1975 was this put into OC 7 or master ?

LukasReschke commented 9 years ago

@PVince81 This is stable7, but unfortunately the regular ZIP-mode added has still problems with OS X. See https://github.com/McNetic/PHPZipStreamer/issues/16

DeepDiver1975 commented 9 years ago

question is how long we are willing to wait ... or shall we dive into this on our own?

LukasReschke commented 9 years ago

I already tried. - It's a world of pain and undocumented OS X behaviour :(

McNetic commented 9 years ago

Just a small update: the remaining issues with OSX finder support are handled here: McNetic/PHPZipStreamer#16. Thanks to @LukasReschke, we got the idea that maybe it could work with with deflate compression enabled. I have a working implementation, but that needs to be merged and unit tests adapted. It depends on pecl_http, however. Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)? I'll have a look into it as soon as my pneumonia is better.

LukasReschke commented 9 years ago

It depends on pecl_http, however. Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)?

Generally speaking it would be way better to avoid additional dependencies as they are not in every case installable and with distributions such as RHEL the fun gets even bigger. - Is there any way that we can avoid that?

Is having that as a dependency in owncloud acceptable (ZipStreamer will still work without it, but will of course fall back to store instead of deflate)

Well - if there is no way around that: It has to be. But before that I guess we should be certain that there is no way around that. Let us know if we can help you somehow. (man power etc…)

karlitschek commented 9 years ago

Yes an additional dependency is a serious problem.

McNetic commented 9 years ago

Well, as far as I can see, standard php has no way to do stream compression. It has at least 3 ways to deflate, but none of them can work on chunks. The only way I found (after quite a search), is in pecl_http. It's no voodoo, it's just that php is obviously badly designed, developed and maintained in some ways. An alternative might be to implement deflate in php itself - but I can't say anything on performance there. It might be fast, perhaps at least for minimum compression, or it might be totally slow. But let's wait and see if it solves the problem at all.

McNetic commented 9 years ago

PS: I don't know if anyone already did, but I reported the problem with apply bug reporter, also. It's really a shame they are successful in telling people they build the best technology.

DeepDiver1975 commented 9 years ago

Upstream project is still working on this issue -> OC8.1

owncloud / core