search

spider-rs / spider

A web crawler and scraper for Rust
https://spider.cloud
MIT License
1.11k stars 96 forks source link

Store referring links #199

Closed LeoDog896 closed 3 months ago

LeoDog896 commented 3 months ago

If spider makes a crawl from one page to the next, there should be an optional feature with an expected performance cost that lets the page store its referrers.

LeoDog896 commented 3 months ago

There is already links.