Open fonty422 opened 3 years ago
Try this. This is also implemented in RocksDB. A python on-disk storage. Great ergonomics.
https://github.com/Congyuwang/RocksDict
Store any python object on-disk also.
You don't need to compile. Just pip install rocksdict
. It is pre-compiled.
from rocksdict import Rdict
import numpy as np
import pandas as pd
path = str("./test_dict")
# create a Rdict with default options at `path`
db = Rdict(path)
db[1.0] = 1
db[1] = 1.0
db["huge integer"] = 2343546543243564534233536434567543
db["good"] = True
db["bad"] = False
db["bytes"] = b"bytes"
db["this is a list"] = [1, 2, 3]
db["store a dict"] = {0: 1}
db[b"numpy"] = np.array([1, 2, 3])
db["a table"] = pd.DataFrame({"a": [1, 2], "b": [2, 1]})
# close Rdict
db.close()
# reopen Rdict from disk
db = Rdict(path)
assert db[1.0] == 1
assert db[1] == 1.0
assert db["huge integer"] == 2343546543243564534233536434567543
assert db["good"] == True
assert db["bad"] == False
assert db["bytes"] == b"bytes"
assert db["this is a list"] == [1, 2, 3]
assert db["store a dict"] == {0: 1}
assert np.all(db[b"numpy"] == np.array([1, 2, 3]))
assert np.all(db["a table"] == pd.DataFrame({"a": [1, 2], "b": [2, 1]}))
# iterate through all elements
for k, v in db.items():
print(f"{k} -> {v}")
# batch get:
print(db[["good", "bad", 1.0]])
# [True, False, 1]
# delete Rdict from dict
db.close()
Rdict.destroy(path)
I'm trying to use Faust, but it fails to correctly set up the rocksdb. Unless I'm missing something obvious, here's the link to the issue I raised for Faust.
We tried to download the source too and then run the setup.py directly, even tried removing the checks for
Wextra
andWconversion
(the two that seem to fail) and it then fails for other reasons.I don't have Anaconda installed (this prove to be problematic with previously installed python and the jupyter notebook) so can't use that either. Any assistance would be greatly appreciated.