search

piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)
MIT License
3.2k stars 383 forks source link

Unable to read gz files on s3 #12

Closed coreyhuinker closed 8 years ago

coreyhuinker commented 9 years ago

It reads them, but it the data remains compressed, thus defeating line iteration.

piskvorky commented 9 years ago

Transparent (de)compression is only supported for local files now.

Should be possible to do it transparently for S3 files too, using Python's zlib from compressed stream processing. Let me know if you want to tackle this -- a low-hanging, extremely useful feature!

ghost commented 9 years ago

+1 for this issue, I have some example code here: https://gist.github.com/brianmingus/a47f26760d244ba7e9d1

asieira commented 9 years ago

It's interesting how smart_open is evolving into something similar for the Python world to what Apache Commons VFS is to the JVM world. Maybe there's some inspiration of the abstractions they used to build something more general.

asieira commented 9 years ago

This is useful for me as well, as soon as https://github.com/piskvorky/smart_open/pull/38 gets merged I could write a PR for this one as well.

piskvorky commented 9 years ago

Thanks for the link @asieira -- I didn't know about VFS.

I'm all for learning from other people's mistakes -- what abstractions and designs in particular do you think would be useful?

asieira commented 9 years ago

The first thing they did was to create the abstraction for a file system, not opening a single file. Going the single file route as smart_open has gone so far is great for formats like xz, gzip and bzip... mas if you want to handle archives that themselves contain internal structure (like zip, tar, etc) that won't work as well. So in VFS, opening a ZIP file is akin to traversing a virtual folder to access the content inside.

Plus, they worked in an arbitrary number of layers. You can build a URI like gz://zip://ftp://ftp.example.com/file/blah.zip!/zipfolder1/file.gz and access a GZIP file, inside a ZIP file, read from an FTP server.

Plus, it's an architecture that can be extended. They defined a set of abstract classes that you can implement to define a new type of filesystem in addition to the built-in ones.

All great ideas, but maybe too complicated for smart_open and worthy of a separate independent project that emulates those ideas in Python.

At the very least, thinking of those ideas I would be inspired to implement support for gzip and bzip compression / decompression in a way that is general for all supported file types in smart_open.

AndreaCrotti commented 8 years ago

+1 on this, I might be able to implement it soon since we'll probably need it..thanks

piskvorky commented 8 years ago

Sounds great @AndreaCrotti ... that would be really useful!

mpenkov commented 8 years ago

This seems to work pretty well as a work-around: https://github.com/commoncrawl/gzipstream

piskvorky commented 8 years ago

Sounds good, thanks for the link @mpenkov ! Can you implement this in a PR?

Depending on how tricky gzipstream is to install and how well supported it is, we could either add it to requirements (if easy), or make it optional (if difficult), or even bundle it inside smart_open directly (license permitting).

@tmylk great intro task?

mpenkov commented 8 years ago

gzipstream doesn't really have any external requirements (just io and zlib) so it shouldn't be hard to install.

@piskvorky I could in theory, but I'm working on something else right now. If this doesn't get assigned to anyone by the time I'm free, I'll have a look at it.

On Fri, Jun 3, 2016 at 2:14 PM Radim Řehůřek notifications@github.com wrote:

Sounds good, thanks for the link @mpenkov https://github.com/mpenkov ! Can you implement this in a PR?

Depending on how tricky gzipstream is to install and how well supported it is, we could either add it to requirements (if easy), or make it optional (if difficult).

@tmylk https://github.com/tmylk great intro task?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/piskvorky/smart_open/issues/12#issuecomment-223553157, or mute the thread https://github.com/notifications/unsubscribe/ABDOVAd8BBNjHNADuBXu-XMsQGNCo_Tnks5qIAyDgaJpZM4Dgi3T .

mpenkov commented 8 years ago

@piskvorky OK, I'm looking into this now.

mpenkov commented 8 years ago

@piskvorky @tmylk I think we can close this now. 78c461e resolved this.

tmylk commented 8 years ago

Thanks @mpenkov ! Closing now

yg37 commented 7 years ago

It seems that smartopen can read a gzip file from s3 using an url, but not using key. Is that the case?