search

BruceDone / awesome-crawler

A collection of awesome web crawler,spider in different languages
MIT License
6.5k stars 709 forks source link

Web-Crawler in Rust #48

Open tclaus opened 5 years ago

tclaus commented 5 years ago

Here is my crawler made in Rust. Supports seed database, adds external links to database, simple parsing and adding to elastic search.

https://github.com/tclaus/web_crawler

let4be commented 2 years ago

Kindly please add mine It's a broad web crawler written in Rust and Redis, with heavy focus on performance, scalability and politeness (almost touching 10 gbit/s on 96 core c5.metal) https://github.com/let4be/crusty