Problems with Zip Files after upgrade to 7.0.0

[maddox-DX]

Steps to reproduce

1. Login und go to files
2. select a couple of files download the generated zip file
3. try to decompress with Total Commander

   Expected behaviour

Files should be extracted

Actual behaviour

Total Commander gives CRC warnings

Server configuration

Operating system: debian 7

Web server: apache 2.2 Database: postgres 9.1 PHP version: 5.4 ownCloud version: (see ownCloud admin page) 7.0.0 Updated from an older ownCloud or fresh install: From 6.0.4

I got an hint from the Total Commander forum: http://ghisler.ch/board/viewtopic.php?t=40766 (in german)

The linked forum post says that the generated zip has two issues:

• the content is not compressed
• the file entries in the zip file don't provide valid crc data (and thats why total commander complains).

[karlitschek]

ok. but we need testing of all the newer os x version. this is unfortunately often incompatible between tar versions. also tar has problems with different charsets, path lengths and so on.

[DeepDiver1975]

also tar has problems with different charsets, path lengths and so on.

hmmm ... is there any archive format which works properly? unbelievable ...

[karlitschek]

Yes. I thought that something that old and stable as tar always works. unfortunately this is more fragile then I thought. but if this is only used on OS X then this can work with proper testing.

[raimund-schluessler]

This sounds like the problems with ZIP unpacking in Total Commander which still does not work (oC 8.1.1 and TotalCommander 8.51a) won't be solved? Is this right?

[LukasReschke]

This sounds like the problems with ZIP unpacking in Total Commander which still does not work (oC 8.1.1 and TotalCommander 8.51a) won't be solved? Is this right?

Can you please file this at https://github.com/McNetic/PHPZipStreamer/issues?

[cdamken]

We use zip files now because it has better compatibility across systems.

How about using RAR? its second in popularity to .zip files and multi-platform.

[cdamken]

One of our users suggested to either make it an option to switch streaming zip files on or off, or use non-streaming ZIP files for files <4 GB

[DeepDiver1975]

One of our users suggested to either make it an option to switch streaming zip files on or off, or use non-streaming ZIP files for files <4 GB

This is a nightmare when it comes down to server memory foot print and other server resources.

[DeepDiver1975]

How about using RAR?

afaik not nativly supported on windows

[PVince81]

Please note that in 8.2 there will be an actions dropdown: this means it would be possible to allow multiple download formats in the menu without cluttering the UI too much:

In that case we might want to have such dropdown in the public link page too when clicking on the button on the top right.

But this means users need to know what to choose.

That was just a side-note :smile:

[cdamken]

afaik not nativly supported on windows

You are right, and the recommendation from Microsoft: Windows does not natively support RAR files, but there are third-party programs that do. To extract files from a compressed RAR file, you need to use a program that is designed to work with RAR files. Ref: http://windows.microsoft.com/en-us/windows/what-is-rar-file#1TC=windows-7

and zip is not native in Linux either: Linux has both zip and unzip program. By default, these utilities are not installed. You can install zip/unzip tools from the shell prompt. Ref:http://www.cyberciti.biz/tips/how-can-i-zipping-and-unzipping-files-under-linux.html

IMO, lets provide a solution with a well know format that can be open in all the devices that we support.

[butonic]

@cdamken Installing third party software defeats the purpose. most linux desktops come with a ui tool that handles zip. gnome file-roller certainly does. Windows zip support is slow as hell ... but works. OSX is the problem. Using your logic, they can just install The Unarchiver. ZIP already is a well known format.

[cdamken]

I was only telling about rar because someone pointed tar too.

@butonic you are right, lets only concentrate in zip and mac in this issue.

[butonic]

@cdamken The idea is to detect OSX and only then use tar.

[jnweiger]

In https://github.com/McNetic/PHPZipStreamer/issues/16#issuecomment-94475010 @McNetic indicates a solution. Added calls to provide crc() and size(). If I understand correctly, this should make the ZipStreamer OSX compatible. But that issue was closed wontfix. :-(

[cdamken]

@butonic the customers wants to have zip and not tar files. I already suggested "The Unarchiver" and many other workarounds.

[butonic]

@cdamken Are you sure he wants zip? Or does he just want to be able to open the downloaded archive with the native osx application. Those are two different things and tar would allow that.

[cdamken]

@butonic I will ask all of them, I'll write again with their expectations.

[MTRichards]

Since this also affects another customer now, we need to get to something that works for Mac. The key is that it work natively on Mac, simple click and download. Whether tar or zip doesn't matter, as long as it works on Mac natively. The specific archive format should be all but invisible to the end user.

Priorities: 1) get some form of compression working seamlessly on all Macs so the product works 2) maybe worry about which format this customer specifically wants

[DeepDiver1975]

So we go with tar and autodetection of the client platform.

@MTRichards would it be enough to have a solution in the scope of 8.2 or are you seeking for backport to 8.1 or even earlier versions?

background: the affected code base will change in the scope of 8.2 - backport would mean: implement it again.

[MTRichards]

Well, unfortunately, we have a lot of customers (including our very own instance) where this isn't working properly. Since there are a lot of open tickets, and people are staying on 6 because of this, and we are dropping 6 in 2 months, and 8.2 isn't out until November....I don't see an alternative to at least 1 backport. I am OK if we only backport to 8.0, but we need to do something to resolve the bug reports if we can.

[karlitschek]

Hey guys. I don't think we want to support different format for the user or admin to choose. the user only wants to have exactly one format that works. Maybe we use zip streamer for all platforms except mac where we use the old zip code or tar? But al transparently and automatic without any options please.

[MTRichards]

But al transparently and automatic without any options please.

YES!

[DeepDiver1975]

But al transparently and automatic without any options please.

YES!

This was the idea from the beginning of the tar discussion: Detect the client platform and if osx send them a tar.

Just in case this got lost: https://github.com/owncloud/core/issues/17930#issuecomment-125590996

[cdamken]

@butonic

@cdamken Are you sure he wants zip?

Confirmed: They want zip! all other solution is just a workaround.

[MorrisJobke]

They want zip! all other solution is just a workaround.

And zip is broken on Mac OS X ... so for files bigger than 4GB this will never work except Apple fixes it :(

[MTRichards]

We can't have folks unclear on the details making blanket statements like "it is a workaround". Sure, it is a workaround for APPLE. End of Story.

We are doing .tar first so it works. @DeepDiver1975 already slated .tar, done.

[karlitschek]

yep. we do what @MTRichards and @DeepDiver1975 proposed. done

[McNetic]

@raimund-schluessler: The problems with TotalCommander were indeed an (unrelated) bug in PHPZipStreamer and are fixed in the current development tree, which is up to now not bundled with owncloud. So, it will work in TotalComander in the future.

[raimund-schluessler]

@McNetic Great, thank you!

[michaelstingl]

@MorrisJobke Will 7.0.9 also generate TAR?

[MorrisJobke]

@MorrisJobke Will 7.0.9 also generate TAR?

Currently there isn't even a solution for master.

[DeepDiver1975]

Gentlemen hold your horses. I even dont know if this can make it into 8.2

[McNetic]

Just to make sure I am not longer an obstacle in getting fixes and improvements into this feature, I just did a release (v0.7) of ZipStreamer, which fixes all known issues except of course the bugs in OSX. I think this version could be bundled with Owncloud as soon as possible.

I know the release fails two tests in Travis-CI - the test work when run locally on any of my machines, so the problem seems to be related to the travis environment. The failing tests also only include those using pecl_http (which is only optional in using ZipStreamer), so it should not be a problem.

As already pointed out in the past, there is a small API change in ZipStreamer since the last release. It should be pretty straightforward to adopt. If you need any help or I forgot anything in doing my release, please tell me.

[ghost]

@DeepDiver1975 check

[DeepDiver1975]

9.0 no time left from my pov. In addition the tar stream code looks evil.

[gig13]

@bboule SF 3561

[McNetic]

I created (hopefully correct) pull requests to import the changes in owncloud/core#19018 and owncloud/3rdparty#196. This also enables zipstreamer to be loaded and updated via composer.

[DeepDiver1975]

I created (hopefully correct) pull requests to import the changes in owncloud/core#19018 and owncloud/3rdparty#196. This also enables zipstreamer to be loaded and updated via composer.

Awesome - THX a lot!

[MTRichards]

Oooh...does this fix zip on Mac?!? (waiting for a party here!)

[DeepDiver1975]

Oooh...does this fix zip on Mac?!? (waiting for a party here!)

no :crying_cat_face:

[MTRichards]

[McNetic]

Can.. not... resist...: fixing zip on Mac has to be done by Apple. Luckily, you'll get an OS update within the next days/weeks. Unfortunately, that simple fix for a years old bug will probably not be included :-(.

Back to topic: As already pointed out earlier, this bug report is technically not about the OSX issues, so I suggest closing this one when the pull request is properly closed and the actual issue in the original bug report is thus fixed. There is a seperate bug report (#13669) for the OSX issues, which I suggest should be used for the further OSX discussion.

[michaelstingl]

@McNetic Probably not fixed in Mac OS X El Capitan :crying_cat_face: :crying_cat_face: :

(Version 10.11 Beta 8)

[karlitschek]

The problem is on Apples side. Definitely! For sure! Now that we have this clarified let me also say that the mission of ownCloud is to provide awesome user experience for our users. So we have to build a workaround. Unfortunately if we care about our users. Which we do.

[DeepDiver1975]

Which we do.

maybe I can come up with a PR to introduce the tar stream - needs some coding ... https://github.com/owncloud/core/issues/10001#issuecomment-139169447

[ghost]

@DeepDiver1975 Any update? This was just raised up as a priority for the Mac

[DeepDiver1975]

this is @VicDeo's current WIP PR to get this fixed #19318

[DeepDiver1975]

fixed by https://github.com/owncloud/core/pull/19318

[cdamken]

AWESOME!!! Thanks guys!! You made my day!!