search

azukaar / Cosmos-Server

☁️ The Most Secure and Easy Selfhosted Home Server. Take control of your data and privacy without sacrificing security and stability (Authentication, anti-DDOS, anti-bot)
https://cosmos-cloud.io
Other
2.95k stars 105 forks source link

[FEAT]: robots.txt per routes #228

Open readtedium opened 3 months ago

readtedium commented 3 months ago

What happened?

I’m attempting to set a robots.txt on a Ghost server I have running on my domain, but nothing is working, and I can’t figure out why. No matter what I do, it remains

User-agent: *
Disallow: /

I would like this server to be accessible to search engines, but I cannot set it. I see a recent change to robots.txt was made to Cosmos that produces the same output and want to confirm this is not the cause.

What should have happened?

When I uploaded a robots.txt file to my Ghost theme, it should have loaded, it did not. It may be related to Cosmos, not Ghost.

How to reproduce the bug?

  1. Load a Ghost server
  2. Upload a theme with a dedicated robots.txt file
  3. Load robots.txt
  4. See disallow message

Relevant log output

No response

Other details

I see that a recent feature was added to support robots.txt to prevent the Cosmos server from being visible on search engines, but even after clearing the site’s cache and removing cookies, the old file is still there.

System details

Client:

Server:

readtedium commented 3 months ago

Update: I turned on the “Allow search engines to index your server” feature on Cosmos and cleared my cache in Cloudflare and the correct robots.txt appeared. I turned off that checkmark and cleared my cache again, and the disallow showed back up.

This is really a feature that should be by domain, because there are some sites I want accessible and others I don’t.

azukaar commented 3 months ago

I will probably add a per domain checkbox yes

readtedium commented 3 months ago

Good call—thank you!