search

antchfx / antch

Antch, a fast, powerful and extensible web crawling & scraping framework for Go
MIT License
260 stars 41 forks source link

new middleware: RFPDupeFilter #6

Closed zhengchun closed 6 years ago

zhengchun commented 6 years ago

  1. RFPDupeFilter is for duplicate URLs filter middleware that based on tylertreat's bloom filter

  2. fixed proxy request.