sebiboga / humans-txt

humanstxt.org is an initiative already forgotten with the goal of stating that WE ARE PEOPLE NOT MACHINES. This initiative is about promoting the initiative.
https://sebiboga.github.io/humans-txt/
MIT License
139 stars 7 forks source link

can we remove duplicates? #33

Open sebiboga opened 1 year ago

sebiboga commented 1 year ago

in the DISCORD channel humans-txt every time you check a domain and it contains a humans.txt file, this domain is logged, but if you test several times same domain, we will log same domain multiple times. the aim is to log one domain one time only in DISCORD channel