search

imthaghost / goclone

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
https://goclone.io
MIT License
1.38k stars 294 forks source link

Recursive clone #56

Open W-Lawless opened 1 year ago

W-Lawless commented 1 year ago

I realize this might be dangerous if someone haphazardly attempts WIkipedia or something, but it would be nice to have an -r flag to recursively follow all tags and clone subpages

mesaglio commented 1 year ago

This sounds more like scrapping, no?

W-Lawless commented 1 year ago

This sounds more like scrapping, no?

I apologize, yesterday when I was looking through here and searching github for a tool, I got very limited results for some reason, although now attempting the same search again there are over 17k repos for this. I swapped the scrape keyword for clone trying to find better results.

That being said, is there any repo you recommend for this in particular? I'd like to essentially clone/scrape a subreddit-style website and make a lot of the info there searchable / indexable. A lot of discussion on programming/opsec stuff that could be made much more actionable if I can search it in ways the exceed the standard sidebar search input.

Thanks!