zrashwani / arachnid

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
MIT License
253 stars 60 forks source link

Fixing Breadth-First-Search crawling & some refactoring #34

Closed zrashwani closed 4 years ago