Feature Request: Always use the latest index from commoncrawl.org

tomnomnom / waybackurls

Fetch all the URLs that the Wayback Machine knows about for a domain

3.41k stars 455 forks source link

Feature Request: Always use the latest index from commoncrawl.org #10

Open cybercdh opened 4 years ago

cybercdh commented 4 years ago

Given the code uses a static index for the commoncrawl.org search, I'm wondering if it's possible to identify and use the latest index, at the time of writing this is http://index.commoncrawl.org/CC-MAIN-2019-51

Thanks

@cybercdh