search

HiSPARC / pysparc

HiSPARC DAQ implemented in Python
GNU General Public License v3.0
3 stars 0 forks source link

Use LevelDB for storing events #12

Open davidfokkema opened 9 years ago

davidfokkema commented 9 years ago

If the StorageManager needs to keep a lot of events in temporary storage, Redis will fill up all memory and more-or-less crash the Pi. I looked into a lot of Redis-like storage solutions, since I like the high-level methods (like incrementing the counters, popping from the front of a list, etc). I tried SSDB (does not support deleting hashes), Redis-NDS (very little activity), Edis (very little activity), Redis-LevelDB (does not know about hmset or hincrby) and Ardb (hangs on a simple storage test using more than one client).

I'll try keeping Redis, but storing the pickled events using LevelDB.

davidfokkema commented 7 years ago

Or ZODB - a native object database for Python?

davidfokkema commented 7 years ago

Or Vedis? Embeddable Redis, thread-safe, with Python bindings.

davidfokkema commented 6 years ago

Nope. No Vedis. Vedis does not have a rpush command, and does lpush in the wrong order. It also has some longstanding bugs...

davidfokkema commented 6 years ago

SSDB has its own protocol, which does allow for deleting hashes. Then, we have everything in one database. Otherwise, keep using Redis for the queue, and use leveldb (or rocksdb) for the events. Two stores...

davidfokkema commented 6 years ago

Hmmm... ArangoDB or MongoDB both have arrays. ArangoDB is not in Stretch, but MongoDB is.

davidfokkema commented 6 years ago

FaunaDB has arrays. It's from "the team that scaled Twitter".