Max retries exceeded with HTTP->HTTPS redirects

[dragonpaw]

I've got a moderately complex setup that I'm trying to migrate from nginx to Caddy2, with redirects and regex and fun stuff like that. So of course I want to run this thing locally and double check the config before putting it in place in production. Unfortunately, this kinda seems impossible with the https features as it is today.

Given this Dockerfile:

FROM caddy/caddy:2.0.0-beta.19-alpine
COPY Caddyfile /etc/caddy/Caddyfile
COPY sites /sites

And a Caddyfile that looks a little like:

{
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    email ash@example.com
    admin off
    local_certs
    debug
}
example.com {
   @foo {
       path_regexp a ^/~ash(.*)
    }
    redir @foo http://ash.example.com{http.regexp.a.1} permanent
    root * /sites/example.com
    file_server
}
[More domains...]

I'd want to be able to do a lot of testing on the redirects and such before, but, using httpie for local test:

$ http http://localhost:80/~ash/foo Host:example.com--verbose
GET /~ash/foo HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: example.com
User-Agent: HTTPie/0.9.8

HTTP/1.1 308 Permanent Redirect
Connection: close
Content-Length: 0
Date: Sun, 29 Mar 2020 00:25:52 GMT
Location: https://example.com/~ash/foo
Server: Caddy

$ http https://localhost/~ash/foo Host:example.com --verbose
http: error: SSLError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /~ash/foo (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1076)'))) while doing GET request to URL: https://localhost/~ash/foo

So I can't do the tests over http, since it always redirects to https, even in debug mode, and this doesn't seem able to be disabled. And trying using https gives some error that doesn't seem to have anything to do with local certificates. And I can't use this in production where the hostnames actually route to the host, because I can't test it.

I'm really not sure how to make progress at this point.

[mholt]

To clarify: Debug mode doesn't disable auto HTTPS redirects -- it has nothing to do with them -- it just enables DEBUG level logging (which there isn't too much of at this point, but I'm sure more will be added with time).

What exactly are you trying to test? What's the problem here, exactly? Your config is served over HTTPS, but redirects to HTTP, I assume for a host that is served by the same config, hence the loop.

I'll need more info in order to be helpful, since just saying "some error" and using clipped or redacted configs isn't going to get us anywhere. :)

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1:

I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.

[sarge]

@dragonpaw something that has been added recently is the ability to do some internal integration testing.

https://github.com/caddyserver/caddy/blob/178ba024fea4db2b91fd159da629f0a8588f119a/caddytest/integration/caddyfile_test.go#L29-L52

This will work with public hostnames, internally it will dial localhost instead of the public dns ip address. https://github.com/caddyserver/caddy/blob/178ba024fea4db2b91fd159da629f0a8588f119a/caddytest/caddytest.go#L222-L226

This will not however work with ACME servers but with a small modification to your Caddyfile you could use caddy's new internal CA (smallstep) to issue certificates for you or you can use locally made self signed certs (the caddytest.Assert... methods ignore certificate verification).

If you are comfortable running some go code.

func TestRespondHttps(t *testing.T) {

    // arrange
    caddytest.InitServer(t, ` 
  {
    http_port     9080
    https_port    9443
  }

  www.cnn.com:9443 {
  # choose your cert management options (tls internal is very cool)
  #  tls internal - load via the internal cert manager
  #  tls /a.caddy.localhost.crt /a.caddy.localhost.key - load using the static certs shipped with the caddytests

    respond /version 200 {
      body "hello from www.cnn.com"
    }   
    }
  `, "caddyfile")

    // act and assert
    caddytest.AssertGetResponse(t, "https://www.cnn.com:9443/version", 200, "hello from www.cnn.com")
}

Finally this is a new testing approach which has few bugs on Windows, if you are a windows user your experiences could help with #3191

Let me know if this is useful.

[dragonpaw]

To clarify: Debug mode doesn't disable auto HTTPS redirects -- it has nothing to do with them -- it just enables DEBUG level logging (which there isn't too much of at this point, but I'm sure more will be added with time).

I didn't expect it to. I was hoping it might add interesting info to the logs. It didn't, but that's a different issue.

What exactly are you trying to test? What's the problem here, exactly? Your config is served over HTTPS, but redirects to HTTP, I assume for a host that is served by the same config, hence the loop.

I am expecting to be able to 100% verify that a configuration under test behaves properly before I let it within 10 miles of production. There's absolutely zero chance that I will deploy an http config, that I haven't tested. Running an httpd within a container and sending it http requests with appropriate Host headers is a pretty good way to be sure that it will behave as desired once it's in use. To that end, I should be able to configure it for, as in this example, to redirect 'example.com/~ash/foo' to 'ash.example.com/foo`. I wish to be able to verify that the config I has will reliably do that, long, long before it's anywhere near production hosts. If I cannot confirm that, I will consider the service unsuitable for production use, and never use it. I'm not about to try to debug httpd configs in production when I can do so before then.

Perhaps this is a philosophical thing. But for me, as a long-time devops engineer, software needs to be able to be tested before it reaches production. So I'm looking for help in finding out if caddy is appropriate for me in this way.

I'll need more info in order to be helpful, since just saying "some error" and using clipped or redacted configs isn't going to get us anywhere. :)

That's cool, but neither of those are true. The config wasn't redacted, and the error message was exactly pasted. (As seen in the last line of the 3rd code block: http: error: SSLError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /~ash/foo (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1076)'))) while doing GET request to URL: https://localhost/~ash/foo)

I referred to the error as "some error" on the text directly below the error.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

Cool, so the config I provided will do exactly that. It's 100% reproducible with the information I've provided. I can even upload the resulting docker container someplace if you like. I shortened the config enough to prove it was reproducible, which is I think maybe what you incorrectly labeled as 'redacting'.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍

The only thing I didn't give before is the version of docker I was using for the build. But sure...

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

Docker 2.2.0.4 under Windows 10-64,

1b. Caddy version (run caddy version or paste commit SHA)

hub.docker.com: 2.0.0-beta.19-alpine
caddy version: v2.0.0-beta.19 h1:6kbQ5jf/lWjD+o3uuq7rnfrvw+x5UU3tuwGpZsLKr7M=

1c. Go version (if building Caddy from source; run go version)

Not built from source.

2. Description

2a. What happens (briefly explain what is wrong)

1. No way to test config outside of production, due to overtly aggressive https settings that don't seem to be able to be disabled.
2. SSL errors when running.

2b. Why it's a bug (if it's not obvious)

Because opinionated old SREs won't put services in production if they can't be put under test. So if I can't bounce a http request off of it with appropriate Host headers, and see the redirects and content I expect to, there's no way I'll ever let it be used by real clients.

2c. Log output

$ http http://localhost:80/~ash/foo Host:example.com--verbose
GET /~ash/foo HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: example.com
User-Agent: HTTPie/0.9.8

HTTP/1.1 308 Permanent Redirect
Connection: close
Content-Length: 0
Date: Sun, 29 Mar 2020 00:25:52 GMT
Location: https://example.com/~ash/foo
Server: Caddy

$ http https://localhost/~ash/foo Host:example.com --verbose
http: error: SSLError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /~ash/foo (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1076)'))) while doing GET request to URL: https://localhost/~ash/foo

2d. Workaround(s)

Can't find any.

2e. Relevant links

https://httpie.org/ https://hub.docker.com/r/caddy/caddy

[sarge]

@dragonpaw I believe you have stumbled on this bug in httpie from 2015 https://github.com/jakubroztocil/httpie/issues/414

As per their issue the recommendation is to edit your /etc/hosts file

http --verify=no https://example.com/~ash/foo --verbose
GET /~ash/foo HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: example.com
User-Agent: HTTPie/0.9.4

HTTP/1.1 301 Moved Permanently
Content-Length: 0
Date: Sun, 29 Mar 2020 06:26:26 GMT
Location: http://ash.example.com/foo
Server: Caddy

## also with curl
curl -i https://example.com/~ash/foo -k
HTTP/2 301 
location: http://ash.example.com/foo
server: Caddy
content-length: 0
date: Sun, 29 Mar 2020 06:16:54 GMT

Btw here is an integration test for you (no hosts name changes needed)

func TestRespondHttps(t *testing.T) {

    // arrange
    caddytest.InitServer(t, ` 
  {
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    email ash@example.com
    admin off
    local_certs   
    debug
    http_port     9080
    https_port    9443
  }

  example.com {
     @foo {
         path_regexp a ^/~ash(.*)
      }
      redir @foo http://ash.example.com{http.regexp.a.1} permanent
      root * /sites/example.com
      file_server
  }

  `, "caddyfile")

    // act and assert
    caddytest.AssertRedirect(t, "https://example.com:9443/~ash/foo", "http://ash.example.com/foo", 301)
}

[dragonpaw]

Good find on the httpie bug! Thank you.

Given that, I think what would have helped in this case would be two things:

1. Some way to disable the force-ssl redirects such that I could test the config and would have found out the ssl issue was unrelated and client-site. (And would be able to test the redirects independent of ssl config.)
2. Some kinda of error message or output from caddy that there was a host-header/SNI-host mismatch on the request it had.

[dragonpaw]

3. Some good docs on how to do local testing, and/or running under docker, how to test using curl to verify before deployment of the container.

[sarge]

You can force http by

 http://example.com {
   ...

The error message you were getting was

http: TLS handshake error from [::1]:56538: no certificate available for 'localhost'

But unless you understand how TLS and SNI work that might not mean much. The TLS / SNI processes are quite separate from the hostname matching.

Sure docs on the use of curl would handy for anyone in a SRE role.

[dragonpaw]

Not sure where you saw the 'no certificate ... for localhost' message, as I never got that one.

I do think some documentation and logging improvements would help in these kinds of situations, but as I have enough to continue now, feel free to close the issue if you'd like.

Thank you for your help.