search

gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.26k stars 1.76k forks source link

[Feature Request] InMemoryStorage supports TTL for visited URLs #136

Open gsoec opened 6 years ago

gsoec commented 6 years ago 
type InMemoryStorage struct {
  // TTL
  visitedURLsTTL time.Duration
}

I think handling TTL for visited URLs internally is quite useful, and we can make redisstorage to support TTL as well. @asciimoo WDYT

Roconda commented 5 years ago

There is an open PR which introduces TTLs: https://github.com/gocolly/redisstorage/pull/2