search

sintaxi / surge

CLI for the surge.sh CDN
https://surge.sh
2.86k stars 136 forks source link

`robots.txt` file on surge.sh subdomains #288

Closed m9491400382 closed 6 years ago

m9491400382 commented 6 years ago
  1. Upload your original project to surge. The project should have "robots.txt", of course.
  2. Then, visit your surge.sh website. Add "/robots.txt" to your URL.

Expected result: Your robots.txt data.

Actual result:

User-agent: * Disallow: /

Surge.sh is clearly doing "data tampering".

sintaxi commented 6 years ago

For obvious reasons we don't permit search engine indexing on our own domain with other users content. Using a throwaway account to file this issue does not compel me believe the policy is incorrect. A robots.txt works as expected with a custom domain.

NourddineMaghmouli commented 4 years ago

sorry I am new into this but what you mean by "a robots.txt works as expected with a custom domain", how can I make my website indexed? should I host it as custom domain?

Thanks

Noureddine

metasoarous commented 3 years ago

@NourddineMaghmouli Is sounds like this is what he's saying, yes.