search

rivermont / spidy

The simple, easy to use command line web crawler.
GNU General Public License v3.0
334 stars 69 forks source link

Fixes crawling of relative url #73 and optimizes the process. #76

Closed lukavia closed 3 years ago

lukavia commented 3 years ago

A quick fix to not create wrong relative urls #73 It also optimizes the process to not try to parse images and videos.