search

danthegoodman1 / icedb

An in-process Parquet merge engine for better data warehousing in S3
https://blog.danthegoodman.com/icedb-v3--third-times-the-charm
Other
128 stars 5 forks source link

Concurrent log file download #118

Closed danthegoodman1 closed 2 weeks ago

danthegoodman1 commented 1 year ago

90+% of the download time is waiting for time to first byte (avg 86ms in region on AWS). If we do batches of these and buffer in memory it should be far faster to parse the log

danthegoodman1 commented 2 weeks ago

duplicate of #74