search

LiuXingMing / SinaSpider

新浪微博爬虫（Scrapy、Redis）
3.27k stars 1.52k forks source link

SinaSpider3 去重应该有bug #48

Closed tangbotony closed 7 years ago

tangbotony commented 7 years ago

self.server.setbit(self.key + str(uid / 4000000000), uid % 4000000000, 1)

key = "dupefilter:%s" % int(time.time())

这里key不唯一，去重应该是失败的。

请楼主确认。