search

jaybaird / python-bloomfilter

Scalable Bloom Filter implemented in Python
MIT License
1.62k stars 330 forks source link

how to use bloomfilter set expire time #37

Closed luoqishuai closed 5 years ago

luoqishuai commented 5 years ago

I want to use bloomfilter in scrapy_redis. scrapy_redis has a large number of url need to filter. If i use the bloomfilter for a few weeks. Linux memory will boom.So i want use bloomfilter to filter 7 days(or other time) url.What shou i do? Can you give me some adices?

luoqishuai commented 5 years ago

If you don't understand or have no free time .Please tell me. Thanks.

kleinron commented 5 years ago

I think you need some kind of aging Bloom filter, as described here. I'm not sure if this repo contains such implementation.

luoqishuai commented 5 years ago

Thank you for your help.