Open Uplink036 opened 1 month ago
Most sites have robots.txt, that are how these websites wants you to crawl their website. This is so you don't flood their server and should be respected. However, this is all I know about the topic right now, and probably should read up more.
https://www.seobility.net/en/wiki/Robots.txt?utm_id=8783357192_87472061646&utm_source=google&utm_medium=cpc&utm_cid=8783357192&utm_agid=87472061646&utm_campaign=geoEN-Wiki&utm_dev=c&utm_devicemodel=&utm_mt=e&utm_term=robots%20txt&gad_source=1&gclid=Cj0KCQjw6auyBhDzARIsALIo6v9Pj0W4j5uprgBArI12FwDiGt6X5PzeKaJdbHQFRsfLZPWHtB8XdY0aAtzGEALw_wcB
Uplink036
commented
1 month ago
Most sites have robots.txt, that are how these websites wants you to crawl their website. This is so you don't flood their server and should be respected. However, this is all I know about the topic right now, and probably should read up more.