Support 7-zip compression

[GoogleCodeExporter]

See original suggestion here:
http://groups.google.com/group/duplicati/browse_thread/thread/9bdd78703b6edd73

Duplicati should support 7-zip compression, and use the built-in encryption
methods.

Original issue reported on code.google.com by kenneth....@gmail.com on 30 Jul 2009 at 4:28

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

This is a very much desired feature.

Original comment by anjdreas on 16 Nov 2011 at 9:33

[GoogleCodeExporter]

I would like to add that by using 7-zip library it will also speed up 
encryption speed since 7-zip supports AES-NI 

Original comment by luminoso on 31 Mar 2012 at 2:10

[GoogleCodeExporter]

You do not have to convince us. We would like to add 7zip support as we hope 
for better compression and performance. The problem is that we did not yet find 
a suitable 7zip library that we could use. Either they support the lzma 
compression but not the 7zip file format or they don't work under Linux.

In the source code repository you will find a branch that includes 7zip 
compression but it only works for Windows. :-( We also have a look at 
http://sharpcompress.codeplex.com/ but their 7zip-support is incomplete. Anyone 
who wants to help them to get 7zip-support done?

Original comment by rst...@gmail.com on 31 Mar 2012 at 7:20

[GoogleCodeExporter]

7Zip will also give Multicore -support for compression which will give speed up 
to 2 - 8 times.

Original comment by ahlgren...@gmail.com on 3 May 2012 at 8:29

[GoogleCodeExporter]

Multicore support is only with LZMA2, not regular 7zip (LZMA). There is someone 
working on a C# version of LZMA2, but I\m not sure if it will support multicore 
right away.

Original comment by kenneth@hexad.dk on 10 May 2012 at 9:25

[GoogleCodeExporter]

This issue was updated by revision r1397.

Duplicati 2.0 trunk now uses SharpCompress and allows compression using LZMA 
(and bz2, etc.)
Big thanks to Simon Colmer for working on this!

For now the file format is still zip, but the compression inside is different.
Once SharpCompress supports 7zip write, we can add that easily. Same goes for 
LZMA2 compression.

Original comment by kenneth@hexad.dk on 1 Aug 2012 at 7:51

[GoogleCodeExporter]

All platforms (mac, linux, win) have a command line program to handle 7z 
archives. Would it be possible to use the command line programs instead of a 
pure .net assembly to handle 7z? That way the native code runs the algorithms 
and the optimal speed is achived on each platform. 
Duplicati would have to create the processes to pack the files with e.g. 7z.exe 
on windows and monitor stdout to update the progress. Duplicati would also need 
an command line generator out of the desired option-dialogs.

Could this work? Or am I something missing?

Original comment by tobias.s...@gmail.com on 5 Sep 2012 at 7:48

[GoogleCodeExporter]

Not sure if that could work, but we have made some progress on this issue. 
Currently we have 7z + LZMA working on developer machines and we are now 
turning a working prototype into a clean implementation. We also have some 
performance ideas on our list to speed up the whole compression process 
significantly.

As we do not like to talk about things that are not done, we have not updated 
this issue with our progress so far. But now that you were asking .... :-)

Original comment by rst...@gmail.com on 6 Sep 2012 at 5:42

[GoogleCodeExporter]

I dislike using executables in that way, because there is always versioning 
issues, parameter issues, error reporting issues, input data issues, status 
reporting problems, etc.

The goal is to make everything a stream process, that is file -> diff -> 
compress -> encrypt -> upload, without ever writing to disk. (That is not 
entirely possible because of async uploads.)

For compression, you would need some intermediate storage. AFAIK, you cannot do 
something like:
7z < stdin &
echo "file1" > stdin
echo filedata > stdin
echo EOF > stdin
echo "file2" > stdin
echo filedata > stdin
echo EOF > stdin

So you would need to dump all files in a folder, while preserving the directory 
structure.
Then upon closing the compressor instance, you would then run:
7z "tempfolder"

Yoy may be able to do the pipe thing with tar instead, and then run 7z on the 
tarball, but then you cannot simply extract a file without decompressing the 
entire tarball later on, and you need to seek through the tarball to find the 
file you want.

And for both cases you would not be able to report the current size of the 
volume, making it hard for Duplicati to decide when to create new volumes.

In short: you can do it but it will be difficult, error prone and slow.

We do have the 7z+Zip engine from Simon, and it is added to trunk, and does not 
have any of the problems listed here. It will be included in the 2.0 release. 
If we do not have too many issues, 7z+LZMA2 will also be included in 2.0.

Original comment by kenneth@hexad.dk on 6 Sep 2012 at 8:13

[GoogleCodeExporter]

No I see the problems when using the executables directly. And as I understand 
it there is progress on using a library so there is no need to use the 
executables directly. The library solution will definetelly be better.

Thank you for you explanations.

Original comment by tobias.s...@gmail.com on 6 Sep 2012 at 10:24

[GoogleCodeExporter]

How about FreeArc?

It works with Linux and Windows and has really developed lately.
In contrast to its own format, it also supports 7z.

Original comment by sauli.ki...@kolumbus.fi on 1 Dec 2012 at 8:49

[GoogleCodeExporter]

:-) Thank you for the suggestion, but we need C# libraries. 

In the meantime we have been busy. We got it working under Windows and we are 
currently fixing issues under Mono for Linux and Mac. This is the last todo 
before we can release a test version (command line) with LZMA/7z.

The intermediate results are great: Our test data (ASCII files + compressed 
files + video and audio files) was compressed in about the same time as zip 
would compress it but the archives were 5-10% smaller. Here we had a trade-off 
between performance (LZMA is usually 9 times slower than zip) and compression 
(LZMA can produce up to 30% smaller archives).

This is actually a contribution from Tobias who did a great job. He ported 
LZMA2 to C#, integrated it into Duplicati and optimized the code for 
performance. This is a really great contribution. He did not only spend a lot 
of time with this he also created an archiving solution for Duplicati that is 
top-notch.

The LZMA/7z code is available at http://code.google.com/p/managed-lzma/ 

Original comment by rst...@gmail.com on 1 Dec 2012 at 10:10

[GoogleCodeExporter]

FYI: We've just made available an experimental build of the command line client 
to try this out. http://www.duplicati.com/news/experimentalbuildcommandlineonly

Original comment by rst...@gmail.com on 19 Dec 2012 at 8:34