wxf4150 / esdump

elasticserch dump import export
8 stars 5 forks source link

option to turn off gzip #1

Closed preussal closed 2 years ago

preussal commented 2 years ago

hi, cool project. I am currently using Elasticdump to keep my Elastic safe. And did a few tests.

With Pipe to gunzip

esdump export --es http://localhost:9200 --index shrink-indexname-000650 -o - | gunzip > shrink-indexname-000650
.
.
.
2021/08/14 12:16:12 total exported 2270609 items; total_raw_bytes: 5569.41 MB;the gzip size: 0.00 MB
2021/08/14 12:16:12 export time spend 14m58.560544995s

Normal way

esdump export --es http://localhost:9200 --index shrink-indexname-000650 -o shrink-indexname-000650.gz
.
.
.
2021/08/14 12:22:59 total exported 2270609 items; total_raw_bytes: 5569.41 MB;the gzip size: 391.23 MB
2021/08/14 12:22:59 export time spend 21m45.181175851s

Since I use ZFS with compression here the comparison

Compress Ratio          Real Size               Compress Size           File
##############################################################################
13.25           5578.07 Mbyte           420.95 Mbyte    shrink-indexname-000650
1.00               391.25 Mbyte           391.15 Mbyte    shrink-indexname-000650.gz
13.50           5566.36 Mbyte           412.23 Mbyte    shrink-indexname-000650.json
###############################################################
Compress Ratio          Real Size               Compress Size
###############################################################
9.42            11.27 Gbyte             1.20 Gbyte

it would be possible to build in a switch that you can work without compression. The space saving is minimal with with ZFS and zstd against gzip.

THX Preussal

wxf4150 commented 2 years ago

i have add gzip option . to disable gzip add parameter "--gzip=false" (default true)