search

bmoscon / cryptostore

A scalable storage service for cryptocurrency data
Other
383 stars 138 forks source link

LZ4 decompression failed (ChunkStore known issue): VersionStore usage instead? #51

Closed flamby closed 4 years ago

flamby commented 4 years ago

Hi,

I'm facing the concurrent read & write error with the arctic backend, listed here: https://github.com/man-group/arctic/issues/655 (error below)

LZ4BlockError: Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 465

I remember having tried monkey-patching cryptostore to use Arctic's VersionStore, as it's mentioned as a quick fix in the arctic issue above, but I remember that it broke cryptostore.

So, is the change of arctic store type something possible or planned?

Thanks, and keep the good work!

bmoscon commented 4 years ago

version store will definitely break if you attempt to use it because its not designed for continuous writes like you'd encounter with cryptostore.

How are you encountering the error? Reading while writing alone wouldnt cause the issue, you'd have to be doing multiple writes at the same time, or like someone in the original issue mentions, have a write die mid-write. The error you posted above looks more like an out of memory issue, unless you know you're killing writes mid-write or doing multiple writes on a symbol at once.

flamby commented 4 years ago

Hi @bmoscon

Thanks for the clarification.

I'm encountering the error simply by reading around 850,000 rows out of 7,000,000 rows in the arctic library (using a chunk_range), while cryptostore is writing. But I had it in the past without any filtering.

I don't think it's a memory issue as I monitored it while reproducing the issue, and I had around 64GB of free memory. I don't think I'm doing multiple writes on a symbol at once as well, since my cryptostore configuration is quite simple. See by yourself :

exchanges:
    BITFINEX:
        retries: -1
        trades: [LTC-BTC, ETH-BTC, XRP-BTC, XMR-BTC, DSH-BTC, ETC-BTC, ZEC-BTC, BTC-USD]
    POLONIEX:
        retries: -1
        trades: [LTC-BTC, ETH-BTC, XRP-BTC, XMR-BTC, DASH-BTC, ETC-BTC, ZEC-BTC, BTC-USDT]
[...]
backfill:
    BITFINEX:
        LTC-BTC:
            start: '2017-01-01'
        ETH-BTC:
            start: '2017-01-01'
        XRP-BTC:
            start: '2017-05-19'

I'll investigate at the mongodb level. The iops are quite good (local SSD), so I don't think it's that, but I'll double check it as well.

The arctic issue discussed about mongodb 4.x. As I'm using mongodb 3.6.3, could it help if I upgrade it to latest 4.2 or 4.3 releases?

Thanks

flamby commented 4 years ago

Hi @bmoscon After several tests, i'm now quite sure the issue is due to a mongodb timeout, because my celery workers reading data from arctic don't have a low latency to reach the mongodb database. when running the arctic tasks on the mongodb server, I can't reproduce the issue.

If you have some recommendations regarding mongodb tuning for arctic, I'll be happy to test them.

closing then.

bmoscon commented 4 years ago

@flamby really you just need to tune mongo for generic performance. https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/ has some suggestions or there are various other blog posts. The most nodes you add to your cluster, the better the performance should be. you may also want to shard, etc. Depends on how much you want to invest in the mongo setup