search

lflare / mdathome-golang

An unofficial rewrite of the official MangaDex@Home client, with focus on performance and reliability!
GNU Affero General Public License v3.0
27 stars 8 forks source link

Being able to stop google bots with robot.txt #19

Closed basilelt closed 3 years ago

basilelt commented 3 years ago

https://search.google.com/search-console/welcome?hl=en

The rest is in title.

Typical google scanning : WARN[2021-06-06T23:25:11+02:00] Request from 66.249.66.207 dropped due to invalid token event=dropped reason="invalid token" referer=None remote_addr=66.249.66.207 url_path=/R4q1FBPxf3mE71L3_hM8jScFtUJZmQFs5jNfJwIwmmY4U_V9Lo-O0_HBjr0rUdu9j2fY47BnxMXKmAMH7eVCllW4n5yhzyBWBTbLKqtJ6WYkLAcE7KYt2v6BQRevfd__RrViHVT5zknptuCqV3zplxIccaro8O-BiOqOYd318GpKQTIylH7ZXujMc9zztkKwnliuo_FxFO9nYJILT_XNMKNySp4/data/38897ca7108f905d8c54ca9a949992c3/v3.jpg