search

cclgroupltd / ccl_chromium_reader

(Sometimes partial) Python re-implementations of the technologies involved in reading various data sources in Chrome-esque applications.
MIT License
134 stars 34 forks source link

Snappy decompression performance #15

Open markbaumgarten opened 1 year ago

markbaumgarten commented 1 year ago

Ran a few tests and found that the snappy decompress function is a bit of a bottleneck.

Using python-snappy instead helped me a lot:

....
import snappy
...

def decompress(data: typing.BinaryIO) -> bytes:
    return snappy.uncompress(data.read())
cclgroupltd commented 1 year ago

Hi there, thanks for that. I think last time I checked that module required the snappy C library installed (and was a pain to use on Windows). Do you know if that's still the case?

markbaumgarten commented 1 year ago

I am only using ubuntu....so I dont know..sorry.

/Mark

On Mon, Jan 9, 2023, 17:14 CCL Solutions Group @.***> wrote:

Hi there, thanks for that. I think last time I checked that module required the snappy C library installed (and was a pain to use on Windows). Do you know if that's still the case?

— Reply to this email directly, view it on GitHub https://github.com/cclgroupltd/ccl_chrome_indexeddb/issues/15#issuecomment-1375889973, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUZW2WP5LVDS2KOZGZXHYTWRQ2PNANCNFSM6AAAAAATRGVZ7Q . You are receiving this because you authored the thread.Message ID: @.***>

cclgroupltd commented 1 year ago

Yeah, I've had a look and it's still not exactly easy to install on Windows and requires an external dependency. So I might look at adding a "try import" around the C bindings and fall back to the pure Python version if that fails so we don't end up locking users out.