How to crawl only the suggested site and not external sites

safesploitOrg / doogle

Doogle is a search engine and web crawler which can search indexed websites and images

https://search.safesploit.com/

MIT License

32 stars 16 forks source link

How to crawl only the suggested site and not external sites #27

Open safesploit opened 8 months ago

safesploit commented 8 months ago

Discussed in https://github.com/safesploitOrg/doogle/discussions/24

^{Originally posted by **benzvi8888** March 10, 2024} Seems when I checked the database the crawler go into social link and start index their content - it crawl Facebook, Twitter, LinkedIn and so on - it filled my database with it. is there any way/option/code to limit the crawling to the site itself only?