search

iv-org / infrastructure

MIT License
1 stars 7 forks source link

Remove robots.txt disallow rules on these public services #3

Open unixfox opened 2 years ago

unixfox commented 2 years ago

Completely remove any disallow rules on these websites in order to have the public services indexed and thus easily accessible from normal people:

When I mean completely removing them: https://stackoverflow.com/questions/36405998/is-this-robots-txt-syntax-with-an-empty-disallow-correct#comment77953337_36406046

TheFrenchGhosty commented 2 years ago

https://redirect.invidious.io - https://invidio.us https://api.invidious.io - https://instances.invidio.us

Both of those shouldn't be indexed.

Otherwise #2

unixfox commented 2 years ago

redirect.invidious.io - invidio.us api.invidious.io - instances.invidio.us

Both of those shouldn't be indexed.

Otherwise #2

They should, so that when normal people type "instances invidious" they find the correct link.

TheFrenchGhosty commented 2 years ago

@unixfox the instance list is on the documentation / on instances.invidious.io

unixfox commented 2 years ago

Let's not make normal people confused, normal people want a quick answer, not dig into multiple websites for finding one link.

We want more people to discover invidious, not fewer people. Every step that make normal people life easier is a good step towards adoption.

TheFrenchGhosty commented 2 years ago

This is why "normal people" are sent to instances.invidious.io

The others aren't made for normal people.

unixfox commented 2 years ago

Ok then we remove the disallow rule on instances.invidious.io?

Perflyst commented 2 years ago

I don't think we should index all those pages. What about indexing the following invidious.io - main page about the project instances.invidious.io - instance list for end users docs.invidious.io - docs for end users and admins

I don't think we should index the following: api.invidious.io - not meant for end users todo.invidious.io - just a redir to github git.invidious.io - just a redir to github

I am not sure about the following: redirect.invidious.io - redirect service for end users but rather used for sharing links than clicking on google (?) uptime.invidious.io - relevant for end users but linked on instance page, also external uptimerobot

unixfox commented 2 years ago

Let's index first all the sites we all agree on.

TheFrenchGhosty commented 2 years ago

Agree with with perflyst on having the 3 indexed, the rest is either "pointless" to index (and are linked to on the website), or just not meant to be used by end user.

unixfox commented 2 years ago

Reopening as I don't agree with the other websites.