New Filesystem: ArchiveFS

[GoogleCodeExporter]

ZipFS is very useful, so much so that I think it should be expanded to support 
many more archive formats.

I am the author of python-libarchive, which is a SWIG wrapper for libarchive. 
Libarchive is a great library that supports somewhere around 30 different 
archive file formats (tar, zip, rar, iso, rpm, deb, 7zip, etc).

By wrapping this library, and building a pyFilesystem fs using it, almost any 
archive should be usable via pyFilesystem.

This is what I have done. Attached is a first revision with unit tests 
(gratefully lifted from ZipFS).

At this time, appending is not supported, but this is something I plan to work 
on in python-libarchive, and will push down to this filesystem when ready.

Original issue reported on code.google.com by btimby@gmail.com on 31 Dec 2011 at 5:40

Attachments:

• archivefs.py
• test_archivefs.py

[GoogleCodeExporter]

This would be an excellent addition to the project.

So what platforms does libarchive run on? Is it a linux thing, or does it 
support Mac + Windows as well?

Will come back to this post New Years festivities, once I'm over the hangover.

Original comment by willmcgugan on 31 Dec 2011 at 12:12

• Changed state: Accepted

[GoogleCodeExporter]

libarchive is cross platform, it supports Linux and Windows, but I don't think 
it supports Mac.

That said, I have only tested the SWIG wrapper (python-libarchive) on Linux.

Have a great new years, I already have a couple new revisions of archivefs, so 
let me know when you circle back, and I can provide updated files.

Original comment by btimby@gmail.com on 31 Dec 2011 at 4:20

[GoogleCodeExporter]

I am not sure if I should reply here or not, but a couple other items.

1. I would love a code review as this is my first usage of pyFilesystem (did 
not know it existed a few days ago).

2. I would like to know about the possibility of commit rights so that I can 
manage this filesystem in fs/contrib/archivefs.py & fs/tests/test_archivefs.py.

Just to let you know a bit more about my plans for pyFilesystem, I currently 
use a very similar in-house library for interacting with the file system and 
archives. My goal is to replace the in-house library with pyFilesystem. And of 
course then send my library to the great bitbucket in the sky.

In addition to archive support, I need an ACL system. But I will open a new 
issue for that requirement.

Original comment by btimby@gmail.com on 31 Dec 2011 at 4:43

[GoogleCodeExporter]

I'm having difficulty getting all the requirements for archivefs (Linux). I've 
installed libarchive-dev with aptget, swig and used pip to install 
'python-libarchive'. But when I try to impory libarchive I get:

ImportError: /usr/local/lib/python2.7/dist-packages/libarchive/__libarchive.so: 
undefined symbol: archive_read_support_filter_uu

Any ideas?

Will raise getting commit rights with the other commiters once I've had a 
chance to play with the code. I'd also like to add an opener for archivefs, so 
the command line tools fsls, fstree etc will work with all archives. Now that 
would be cool!

Original comment by willmcgugan on 5 Jan 2012 at 5:03

[GoogleCodeExporter]

It is a bit difficult right now, yes.

I specifically only support libarchive 3+. This is very new code, but
it is the version supporting RAR, which is a format I must support for
my use-case. In other words, I only wrote this extension to get
libarchive with RAR support, so I ignored libarchive 2.

What I ended up doing on my machine was to download libarchive 3 and
build from source (I have only tested on libarchive 3.0.1 at this
point, 3.0.2 was just released).

$ wget http://libarchive.googlecode.com/files/libarchive-3.0.1b.tar.gz
$ tar xzf libarchive-3.0.1b.tar.gz
$ cd libarchive-3.0.1b
$ ./configure
$ make
$ sudo make install

Then I could build the extension like this:

$ wget 
http://python-libarchive.googlecode.com/files/python-libarchive-3.0.1-1.tar.gz
$ tar xzf python-libarchive-3.0.1-1.tar.gz
$ cd python-libarchive-3.0.1-1
$ python setup.py build_ext --include-dirs='libarchive'
--extra-link-args='-l:libarchive.so.11.0.1'
$ sudo python setup.py install

libarchive.so.11.0.1 refers to the libarchive API version, 11.0.1 is
the API version for libarchive 3.0.1. The --extra-link-args ensures
that the extension is linked against the correct version. Otherwise,
it will link against /usr/lib/64/libarchive.so, which will be a
symlink to the "official" version you have installed via your package
manager.

The process is kinda documented below. I am working to make this
easier and smoother. At some point I might support libarchive 2, but
it has a very different API, so that would really required writing two
versions of the extension.

http://code.google.com/p/python-libarchive/wiki/Building

Original comment by btimby@gmail.com on 5 Jan 2012 at 5:43

[GoogleCodeExporter]

That is fine. I will work with you however you prefer.

The new archivefs revision has an ArchiveMountFS, which is a subclass
of MountFS. It automatically notices any archives and mounts them
using ArchiveFS.

Think of it as a JIT mounter for archives.

Original comment by btimby@gmail.com on 5 Jan 2012 at 5:45

[GoogleCodeExporter]

I updated the build instructions on the python-libarchive wiki. I also made a 
new release which should improve the installation procedure.

http://code.google.com/p/python-libarchive/wiki/Building
http://python-libarchive.googlecode.com/files/python-libarchive-3.0.3-1.tar.gz

Can you give it a try and let me know if you are successful? I would like to 
get some feedback on archivefs.

Original comment by btimby@gmail.com on 20 Jan 2012 at 5:32

[GoogleCodeExporter]

Got that building. Looks good. I've added the files to the repos. Would be 
great if you could get it passing with the tests for the other implementations, 
and ideally the threading test cases as well.

Would you still commit rights? BTW what's your real name? I only see your gmail 
handle.

Original comment by willmcgugan on 20 Jan 2012 at 6:14

[GoogleCodeExporter]

Original comment by gc...@loowis.durge.org on 24 Aug 2012 at 10:22

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by gc...@loowis.durge.org on 24 Aug 2012 at 10:23

• Changed state: Started