Support HTTPS for custom domains

alex commented 7 years ago

It'd be great if RTD supported HTTPS when using a custom domain. Once upon a time this would have been difficult and expensive, however nowadays with SNI and Lets Encrypt I think this is possible -- providers like SquareSpace and Wordpress have demonstrated this.

Here's how it could work:

When a new custom domain is added, trigger an attempt to obtain a certificate, triggering the lets encrypt validation flow.
Redirect all /.well-known/* from all RTD sites to a special validation server
On the validation server, serve the correct value for the LE challenge for that domain
Obtain the certificate, and store it somewhere.
(Make sure to repeat every X days so certs don't expire!)
When a request comes in with SNI=foo.com, look up that cert and server it if it's a RTD site

Hopefully that makes sense!

alex commented 7 years ago

Here's SquareSpace's write up of how they did it: https://engineering.squarespace.com/blog/2016/implementing-ssl-tls-for-all-squarespace-sites

alex commented 7 years ago

Here's the Etsy post: https://codeascraft.com/2017/01/31/how-etsy-manages-https-and-ssl-certificates-for-custom-domains-on-pattern/

alex commented 7 years ago

And one more for good measure :-) https://www.digitalgov.gov/2016/09/07/lets-encrypt-those-cnames-shall-we/

alex commented 7 years ago

Just occurred to me that Caddy (https://caddyserver.com/) can be used as an automatically-adding-tls-reverse-proxy.

agjohnson commented 7 years ago

We've been wanting to add letsencrypt support for a while. We have an idea of how it would all come together, and this is on our list of features we'll implement as soon as it makes sense financially. We're hoping sometime in the first half of this year perhaps?

This will probably be an equal amount of effort on the code and on the operations side. We have too much logic built in to nginx currently, we wouldn't swap that out, but perhaps running a secondary service for /.well-known/ serving would be fine. There are a number of issues that we'd have to overcome operationally, such as how to keep our web cluster certs in sync, how nginx handles the certs, etc. These all have solutions, but the changes are still significant.

This all goes back to #328, but I think letsencrypt is the best way we have of solving this problem.

alex commented 7 years ago

Awesome! Hopefully some of the links I dropped in here proved useful.

On Mon, Feb 20, 2017 at 12:38 PM, Anthony notifications@github.com wrote:

We've been wanting to add letsencrypt support for a while. We have an idea of how it would all come together, and this is on our list of features we'll implement as soon as it makes sense financially. We're hoping sometime in the first half of this year perhaps?

This will probably be an equal amount of effort on the code and on the operations side. We have too much logic built in to nginx currently, we wouldn't swap that out, but perhaps running a secondary service for /.well-known/ serving would be fine. There are a number of issues that we'd have to overcome operationally, such as how to keep our web cluster certs in sync, how nginx handles the certs, etc. These all have solutions, but the changes are still significant.

This all goes back to #328 https://github.com/rtfd/readthedocs.org/issues/328, but I think letsencrypt is the best way we have of solving this problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rtfd/readthedocs.org/issues/2652#issuecomment-281140560, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBIqN4m2LUcdCz40O_bqMxzwSxBBLks5rec-cgaJpZM4MFUIF .

-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

rmzelle commented 6 years ago

GitHub seems to be slowly rolling out HTTPS support for GitHub Pages, also using Let's Encrypt. See https://github.com/isaacs/github/issues/156#issuecomment-366543897 and the rest of that thread.

Would love to see HTTPS support for RTD custom domains as well!

rmzelle commented 6 years ago

(GitHub just announced official support for HTTPS for GitHub Pages served at custom domains. See https://twitter.com/github/status/991366832421523456)

davidfischer commented 6 years ago

@rmzelle, this is on our roadmap. I see a 5 step roadmap toward HTTPS everywhere on docs sites on RTD. Here's an outline: https://github.com/rtfd/readthedocs.org/issues/3282#issuecomment-369509347. Here's a PR for step 1: https://github.com/rtfd/readthedocs.org/pull/3987.

If you are interested in getting involved, you can start by reviewing the PRs in this area starting with https://github.com/rtfd/readthedocs.org/pull/3987 or if you want to work on this part (step 3 as I see it), that's fantastic! I am specifically very interested and actively working toward HTTPS everywhere on RTD but I will take all the help I can get.

davidfischer commented 6 years ago

To give a small update here, I'm working with Cloudflare to make this a reality. If all goes well, I'd say it's 2 weeks out.

davidfischer commented 6 years ago

I wouldn't quite call this "done" but as of today Cloudflare is issuing SSL certificates for custom domains hosted by us.

Our CNAME support has been around a long time so there are still a lot of domains pointing to readthedocs.org instead of readthedocs.io. I'm working through some issues but eventually I hope that I can issue certificates for domains with the old setup. In the meantime, pointing the CNAME to readthedocs.io should do it.

There are also folks hosting docs at top level domains (eg. cryptography.io) and I haven't quite worked through how that will work.

alex commented 6 years ago

FWIW, cryptography.io's DNS points at a reverse proxy we run, so that's may be a distinct case. (Let us know if we can be helpful!)

On Tue, Jul 17, 2018 at 4:46 PM David Fischer notifications@github.com wrote:

I wouldn't quite call this "done" but as of today Cloudflare is issuing SSL certificates for custom domains hosted by us.

Our CNAME support https://docs.readthedocs.io/en/latest/alternate_domains.html#cname-support has been around a long time so there are still a lot of domains pointing to readthedocs.org instead of readthedocs.io. I'm working through some issues but eventually I hope that I can issue certificates for domains with the old setup. In the meantime, pointing the CNAME to readthedocs.io should do it.

There are also folks hosting docs at top level domains (eg. cryptography.io) and I haven't quite worked through how that will work.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rtfd/readthedocs.org/issues/2652#issuecomment-405738656, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBISQ-ujiLnpH3JrS7o57aJ14R0FHks5uHlsvgaJpZM4MFUIF .

-- All that is necessary for evil to succeed is for good people to do nothing.

davidfischer commented 6 years ago

cryptography.io's DNS points at a reverse proxy we run, so that's may be a distinct case.

True. I used it as an example and it wasn't an ideal example.

desertkun commented 6 years ago

Apparently having a CNAME to readthedocs.io now results in a Error 1014: CNAME Cross-User Banned if the user itself uses cloudflare too.

I've been using cloudflare to http proxy to readthedocs.io to have custom domain with HTTPS, and because apparently it's now user cloudflare too, my domain is banned.

stsewd commented 6 years ago

@desertkun please see https://github.com/rtfd/readthedocs.org/issues/4395

begriffs commented 6 years ago

I wouldn't quite call this "done" but as of today Cloudflare is issuing SSL certificates for custom domains hosted by us.

Do you have a rough estimate for when http->https redirection will be enabled?

rmzelle commented 6 years ago

In the meantime, pointing the CNAME to readthedocs.io should do it.

It looks like this is the case for our site:

$ dig docs.citationstyles.org CNAME
...
docs.citationstyles.org. 6745   IN  CNAME   readthedocs.io

However, Firefox reports for https://docs.citationstyles.org/en/stable/:

docs.citationstyles.org uses an invalid security certificate. The certificate is only valid for the following names: ssl905239.cloudflaressl.com, *.readthedocs.io, readthedocs.io Error code: SSL_ERROR_BAD_CERT_DOMAIN

astrojuanlu commented 6 years ago

Same issue here:

$ dig docs.poliastro.space CNAME +short
readthedocs.io.

Firefox:

docs.poliastro.space uses an invalid security certificate. The certificate is only valid for the following names: ssl905239.cloudflaressl.com, *.readthedocs.io, readthedocs.io Error code: SSL_ERROR_BAD_CERT_DOMAIN

Chromium:

This server could not prove that it is docs.poliastro.space; its security certificate is from ssl905239.cloudflaressl.com. This may be caused by a misconfiguration or an attacker intercepting your connection.

davidfischer commented 6 years ago

@Juanlu001 @rmzelle Both of these domain names somehow got into a bad state with respect to Cloudflare issuing the certificate. When I manually retried the request (which is supposed to happen automatically) it worked. I don't yet have a good reason for why this happened but both domains now have a working certificate.

astrojuanlu commented 6 years ago

@davidfischer confirmed, thanks!

emodric commented 6 years ago

@davidfischer When subprojects are accessed via https (e.g. https://example.com/projects/subproject/) they redirect to HTTP variant: http://example.com/projects/subproject/en/latest/. Is there something that can be done about that?

davidfischer commented 6 years ago

Yes. I'm working on it. I of course welcome help.

emodric commented 6 years ago

I can test perhaps. I doubt I can be of help any further :)

nijel commented 6 years ago

Same issue happened with docs.weblate.org. I previously proxied the content to get SSL and when switching to readthedocs.io CNAME, I ended up with this. Maybe there was DNS caching issue on the server which still saw old CNAME and thus did not issue the cert?

davidfischer commented 6 years ago

If you just changed the CNAME, it can take up to an hour. Did you just change it? I see your certificate as "pending issuance" so it should be issued shortly.

nijel commented 6 years ago

Yes, I did change it (sorry for not being clear in the previous comment). I've tried to improve docs to cover this situation so that others do not have to ask same question :-). See https://github.com/rtfd/readthedocs.org/pull/4487

davidfischer commented 6 years ago

I'm seeing errors related to docs.weblate.org. It's possible this will be resolved in time, but it looks like you've added a number of DNS records pointing to Read the Docs that are unnecessary:

dig docs.weblate.org A +short
dig docs.weblate.org TXT +short
dig docs.weblate.org MX +short

Only the CNAME -> readthedocs.io is needed.

nijel commented 6 years ago

Just remove +short and you will see these are glue records from your resolver:

$ kdig docs.weblate.org TXT 
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 14700
;; Flags: qr rd ra; QUERY: 1; ANSWER: 3; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; docs.weblate.org.        IN  TXT

;; ANSWER SECTION:
docs.weblate.org.       5904    IN  CNAME   readthedocs.io.
readthedocs.io.         182 IN  TXT "google-site-verification=KNkeKlikmpSoWOQ_xkkuIKXUKrvWivh3hHDKCZhSHb0"
readthedocs.io.         182 IN  TXT "google-site-verification=_9m86KBk0-5kAqCFQmrdr3PIvq562Qtkf6q9kBaiXE0"

;; Received 224 B
;; Time 2018-08-08 07:33:28 CEST
;; From 127.0.0.1@53(UDP) in 1.9 ms
$ kdig docs.weblate.org A
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 16137
;; Flags: qr rd ra; QUERY: 1; ANSWER: 6; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; docs.weblate.org.        IN  A

;; ANSWER SECTION:
docs.weblate.org.       5888    IN  CNAME   readthedocs.io.
readthedocs.io.         300 IN  A   104.18.227.122
readthedocs.io.         300 IN  A   104.18.228.122
readthedocs.io.         300 IN  A   104.18.229.122
readthedocs.io.         300 IN  A   104.18.230.122
readthedocs.io.         300 IN  A   104.18.231.122

;; Received 142 B
;; Time 2018-08-08 07:33:44 CEST
;; From 127.0.0.1@53(UDP) in 7.9 ms
$ kdig docs.weblate.org CNAME
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 48250
;; Flags: qr aa rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; docs.weblate.org.        IN  CNAME

;; ANSWER SECTION:
docs.weblate.org.       5885    IN  CNAME   readthedocs.io.

;; Received 62 B
;; Time 2018-08-08 07:33:47 CEST
;; From 127.0.0.1@53(UDP) in 0.1 ms

Anyway apparently this is not yet ready for switch, I'm reverting to the old proxying setup for now. In case you want to test something, I can switch the DNS back to the readthedocs.io.

davidfischer commented 6 years ago

Anyway apparently this is not yet ready for switch, I'm reverting to the old proxying setup for now.

The cert was successfully issued, finally.

nijel commented 6 years ago

You mean after I changed CNAME back to my own server? DNS caching can be sometimes funny. Thanks it indeed works now.

nijel commented 6 years ago

Also it might be worth mentioning in the docs is need to adjust CAA if domain has one, see https://support.cloudflare.com/hc/en-us/articles/115000310832-Certification-Authority-Authorization-CAA-FAQ

davidfischer commented 6 years ago

Also it might be worth mentioning in the docs is need to adjust CAA if domain has one

Good idea.

ivankravets commented 6 years ago

Do we have any options to make 301 Moved Permanently from HTTP to HTTPS using built-in CNAME SSL provided by Cloudflare?

We can inject JS redirect but it's not good for search engines.

davidfischer commented 6 years ago

@ivankravets, I am going to start with 302s but redirects are coming.

ivankravets commented 6 years ago

@davidfischer Thank you so much for the fast response. Do you have any ETA? Will we have some checkbox in UI/Admin part?

davidfischer commented 6 years ago

I don't have an ETA right now. There will be a checkbox in the admin. That's actually merged already and will go live at the next deploy. See https://github.com/rtfd/readthedocs.org/pull/4483.

davidfischer commented 6 years ago

There will be a checkbox in the admin

There is a checkbox in the admin now.

ivankravets commented 6 years ago

@davidfischer do you have any ideas why it does not redirect from http://docs.platformio.org to https://docs.platformio.org? However, it redirects from "nested" pages.

P.S: I enabled HTTPS for docs.platformio.org

davidfischer commented 6 years ago

There are no redirects in place for custom domains. Right now, all that checkbox does is make all the "view docs" links anywhere on Read the Docs an HTTPS link. The plan is to use that checkbox to do HTTPS redirects but that is not in place today.

omidraha commented 6 years ago

Hi, I configure my domain as below:

Cloudflare:

screenshot_2018-09-18 dns omidraha com or omidraha com s account cloudflare - web performance security

rtd:

screenshot_2018-09-18 edit domains read the docs

But still this address http://omidraha.com is not automatically forced to https://omidraha.com So what extra configuration I need to do to force http://omidraha.com to https://omidraha.com ?

stsewd commented 6 years ago

@omidraha please see https://github.com/rtfd/readthedocs.org/issues/2652#issuecomment-413268346

agjohnson commented 6 years ago

I'm going to close up this issue, as we do now support custom domains through Cloudflare now. If you are hitting any issues with custom domains, be sure to open up a new ticket!

ryanpitts commented 5 years ago

There are no redirects in place for custom domains. Right now, all that checkbox does is make all the "view docs" links anywhere on Read the Docs an HTTPS link. The plan is to use that checkbox to do HTTPS redirects but that is not in place today.

Is there an issue that's tracking work toward automatic HTTPS redirects? Sorry if I missed it, but did a search and didn't turn up something specific. Thanks again for enabling HTTPS support for custom-domain projects, I'm excited to be able to push all traffic that direction for our stuff!

stsewd commented 5 years ago

@ryanpitts https://github.com/rtfd/readthedocs.org/issues/4641 https://github.com/rtfd/readthedocs.org/issues/4135

ryanpitts commented 5 years ago

@stsewd ohhhhhhh. Haha, thank you! Apparently "https" isn't a particularly useful search term across GitHub issues.

readthedocs / readthedocs.org

Support HTTPS for custom domains #2652