search

faust-streaming / faust

Python Stream Processing. A Faust fork
https://faust-streaming.github.io/faust/
Other
1.6k stars 180 forks source link

Table TTL #515

Closed coredump17 closed 10 months ago

coredump17 commented 1 year ago

Hi All,

This is more of a question than an issue - not sure if you have a community channel instead?

i see someone posted on the original faust repo, asking if and how we can expire table entries using TTL in 2018 - https://github.com/robinhood/faust/issues/230 . Has this feature been implemented? This would be handy to prevent table bloat over time, when not using windowed tables.

wbarnha commented 1 year ago

Can't believe I didn't stumble upon this issue sooner. It's a great idea that could benefit many. Looks like rocksdict has support in https://congyuwang.github.io/RocksDict/rocksdict.html#AccessType and if I use python-rocksdb, I need to set wal_ttl_seconds.

I think this is a relatively easy feature to introduce, we should add an option in https://github.com/faust-streaming/faust/blob/master/faust/stores/rocksdb.py. I think this feature may be a good time to cleanup rocksdb.py into two separate files, one for python-rocksdb, and the other for rocksdict, to make organization cleaner/less embarrassing.