search

stackmuncher / stm_server

The server-side code behind stackmuncher.com, a global directory of software developers.
https://stackmuncher.com
GNU Affero General Public License v3.0
2 stars 0 forks source link

NOINDEX #29

Open rimutaka opened 2 years ago

rimutaka commented 2 years ago

The current indexing strategy makes no sense. We should let the search engines index anything they like and then review.

rimutaka commented 2 years ago

The rationale of using NOINDEX on personal profiles is that it's one thing we analyze someone's profile and the other is to make it readily available via a search engine. This feature would need to be controlled by profile owners and should be no-index unless specified otherwise.

Update

The change I made in 8ffe203 is for anonymous profiles. Those are definitely not indexed. GH profiles were indexed and are still indexed. I do not know if it's a good thing or not. The profiles are a gateway to projects. Given how few people complained I'm tempted to leave it as INDEX for now.

rimutaka commented 2 years ago

Update

I added a rule to allow blocked bots to fetch robots.txt. Bing finally picked it up and stopped hammering the site.

image