Open karlicoss opened 1 year ago
Ah yeah, totally down for adding this, falling back to default behaviour if it fails
have been thinking more about this with me adding more formats to browserexport, will probably create a meta-package like you have in exporthelpers that this will have as a dependency
Another relevant thing that may be worth extracting from HPI is a library for accessing compressed stuff https://github.com/karlicoss/kompress/issues/10 I think after a few years unfortunately there isn't anything existing
started extracting kompress stuff here btw https://github.com/karlicoss/kompress -- will add more docs and think if needs any refactoring and then will move HPI and bleanser to use it
looks good
I think the only thing it doesnt meet my usecase for is .gz
files (not .tar.gz
files)
gzip has the same rb
-> rt
problem lzma does
will PR that, would be nice to be able to use that in my tools instead of re-implementing it everywhere
I guess not a super big deal since we use caching, but it does give significant (almost 2x speedups)
Had good success using it for a couple of DALs https://github.com/karlicoss/exporthelpers/blob/804b8afa070d8017ad15710a2a179e71ea60316f/dal_helper.py#L140-L171 (made it an optional dependency for backwards compatibility since
ijson
involves some binaries which might be unavailable for some platforms)related: https://github.com/seanbreckenridge/google_takeout_parser/issues/40