Caddyfile v1 -> v2 reverse proxy don't work

[DaoYangM]

1. Which version of Caddy are you using (caddy -version)?

commit 8de1a762273323608d9d9080c42057d1814070b6 (HEAD -> v2, tag: v2.0.0-beta10, origin/v2) Author: Matthew Holt mholt@users.noreply.github.com Date: Mon Nov 18 14:22:55 2019 -0700

2. What are you trying to do?

Changing V1 Caddyfile to V2 V1 Caddyfile woked fine BUT V2 reverse_proxy don't work

3. What is your Caddyfile?

V1

https://www.daoyang.fun:443 {
        root /www

        proxy /helloworld https://127.0.0.1:42111 {
                header_upstream Host "www.daoyang.fun"
                header_upstream X-Forwarded-Proto "https"
                insecure_skip_verify
        }
}

V2

https://www.daoyang.fun {
    file_server {
        root /www
        browse
    }    

    reverse_proxy /helloworld 127.0.0.1:36601 {

        header_up Host www.daoyang.fun
        header_up X-Forwarded-Proto https
        transport http {
            tls_insecure_skip_verify
        }
    }
}

I think the two profiles only have proxy upstream of V1 with HTTPS and V2 reverse_proxy without. If I add HTTPS upstream of V2 like this

reverse_proxy /helloworld https://127.0.0.1:36601 {...}

then the service will report an error

2019/11/19 12:32:28.108 ERROR   http.log.error  dial https:: unknown network https:     {"request": {"method": "PUT", "uri": "/helloworld%0A", "proto": "HTTP/2.0", "remote_addr": "27.156.68.2:63751", "host": "www.daoyang.fun", "headers": {"Accept-Encoding": ["identity"], "User-Agent": ["Go-http-client/2.0"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "status": 502, "err_id": "mvuw2eebv", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:330)"}

So I removed HTTPS from V2, and the proxy service did not respond even though there was no error in the service log

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy run --config /www/Caddyfile --adapter caddyfile

5. Please paste any relevant HTTP request(s) here.

6. What did you expect to see?

reverse proxy can work

7. What did you see instead (give full error messages and/or log)?

nothing log and server don't response correctly

8. Why is this a bug, and how do you think this should be fixed?

Can add https protocol in reverse_protocal like V1 version

9. What are you doing to work around the problem in the meantime?

10. Please link to any related issues, pull requests, and/or discussion.

Bonus: What do you use Caddy for? Why did you choose Caddy?

for v2ray h2 proxy

[AbdelmalekIhdene]

Try removing tls_insecure_skip_verify unless your back-end is listening on the TLS port with an HTTPS webserver ready to take in input.

[DaoYangM]

I tried but don't work, and my back-end is http2. V1 Caddyfile also has insecure_skip_verify

[francislavoie]

In the error message, "uri": "/helloworld%0A" looks odd. Why is there a linefeed character there?

[DaoYangM]

@francislavoie It’s really weird, I’ve wrapped it in double quotes, will it be that Caddy2 doesn’t support reverse proxy to https or h2?

[mholt]

The upstreams need to be net dialer addresses (i.e. not URLs, but host:port). To enable proxying via HTTPS (i.e. HTTP + TLS) to an upstream, use:

transport http {
    tls
}

So I think you're just missing that one line. You would thus need:

https://www.daoyang.fun {
    file_server {
        root /www
        browse
    }    

    reverse_proxy /helloworld 127.0.0.1:36601 {

        header_up Host {host}
        header_up X-Forwarded-Proto {proto}
        transport http {
            tls
            tls_insecure_skip_verify
        }
    }
}

[DaoYangM]

@mholt Unfortunately, this still doesn't work and there are still no logs. Below is the latest code I am using.

https://www.daoyang.fun {
    file_server {
        root /www
        browse
    }

    reverse_proxy "/helloworld" {
        to 127.0.0.1:36601
        header_up Host www.daoyang.fun
        header_up X-Forwarded-Proto https
        transport http {
            tls
            tls_insecure_skip_verify
        }
    }
}

[mholt]

@DaoYangM I'm still working on adding logging config to the Caddyfile, can you convert that to JSON and add logging? Use caddy adapt to do that, then just add logging config: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#logging

[DaoYangM]

@mholt Here is the code after converting to json format and add logging configuration

{
    "logging": {
    "logs": {
        "default": {
            "writer": {
             "output": "stdout"
        },
        "encoder": {
            "format": "console"
        },
        "level": "debug",
        "include": []
        }
    }
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "www.daoyang.fun"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "headers": {
                                                        "request": {
                                                            "set": {
                                                                "Host": [
                                                                    "www.daoyang.fun"
                                                                ],
                                                                "X-Forwarded-Proto": [
                                                                    "https"
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    "transport": {
                                                        "protocol": "http",
                                                        "tls": {
                                                            "insecure_skip_verify": true
                                                        }
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "127.0.0.1:36601"
                                                        }
                                                    ]
                                                }
                                            ],
                                            "match": [
                                                {
                                                    "path": [
                                                        "/helloworld"
                                                    ]
                                                }
                                            ]
                                        },
                                        {
                                            "handle": [
                                                {
                                                    "browse": {},
                                                    "handler": "file_server",
                                                    "hide": [
                                                        "Caddyfile"
                                                    ],
                                                    "root": "/www"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

then I got logs

1.5742153105701478e+09  debug   http.handlers.reverse_proxy     upstream roundtrip      {"request": {"method": "PUT", "uri": "/helloworld", "proto": "HTTP/2.0", "remote_addr": "27.156.68.2:50525", "host": "www.daoyang.fun", "headers": {"User-Agent": ["Go-http-client/2.0"], "X-Forwarded-For": ["27.156.68.2"], "Accept-Encoding": ["identity"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "resumed": 4865, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "headers": {"Cache-Control": ["no-store"], "Date": ["Wed, 20 Nov 2019 02:01:50 GMT"]}, "status": 200}
1.5742153686376066e+09  debug   http.handlers.reverse_proxy     upstream roundtrip      {"request": {"method": "PUT", "uri": "/helloworld", "proto": "HTTP/2.0", "remote_addr": "112.49.83.128:14330", "host": "www.daoyang.fun", "headers": {"Accept-Encoding": ["identity"], "User-Agent": ["Go-http-client/2.0"], "X-Forwarded-For": ["112.49.83.128"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 771, "resumed": 49196, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "headers": {"Cache-Control": ["no-store"], "Date": ["Wed, 20 Nov 2019 02:02:48 GMT"]}, "status": 200}
1.5742153686378298e+09  debug   http.handlers.reverse_proxy     upstream roundtrip      {"request": {"method": "PUT", "uri": "/helloworld", "proto": "HTTP/2.0", "remote_addr": "112.49.83.128:14330", "host": "www.daoyang.fun", "headers": {"Accept-Encoding": ["identity"], "User-Agent": ["Go-http-client/2.0"], "X-Forwarded-For": ["112.49.83.128"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 771, "resumed": 49196, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "headers": {"Cache-Control": ["no-store"], "Date": ["Wed, 20 Nov 2019 02:02:48 GMT"]}, "status": 200}
1.5742153909287667e+09  debug   http.handlers.reverse_proxy     upstream roundtrip      {"request": {"method": "PUT", "uri": "/helloworld", "proto": "HTTP/2.0", "remote_addr": "112.49.83.128:14330", "host": "www.daoyang.fun", "headers": {"User-Agent": ["Go-http-client/2.0"], "X-Forwarded-For": ["112.49.83.128"], "Accept-Encoding": ["identity"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 771, "resumed": 49196, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "headers": {"Date": ["Wed, 20 Nov 2019 02:03:10 GMT"], "Cache-Control": ["no-store"]}, "status": 200}
1.5742154245975034e+09  debug   http.handlers.reverse_proxy     upstream roundtrip      {"request": {"method": "PUT", "uri": "/helloworld", "proto": "HTTP/2.0", "remote_addr": "112.49.83.128:14330", "host": "www.daoyang.fun", "headers": {"Accept-Encoding": ["identity"], "User-Agent": ["Go-http-client/2.0"], "X-Forwarded-For": ["112.49.83.128"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 771, "resumed": 49196, "proto": "h2", "proto_mutual": true, "server_name": "www.daoyang.fun"}}, "headers": {"Cache-Control": ["no-store"], "Date": ["Wed, 20 Nov 2019 02:03:44 GMT"]}, "status": 200}

It looks fine, but it doesn't work and my back-end reverse proxy 127.0.0.1:36601 server is an HTTP2+TLS

[mholt]

@DaoYangM Thanks for the update!

It looks fine, but it doesn't work and my back-end reverse proxy 127.0.0.1:36601 server is an HTTP2+TLS

The upstream requests apparently yield a 200 response, which is an OK status. What do you mean by "doesn't work"?

[DaoYangM]

@mholt It seems that after arriving at the proxy server, there is no proxy back.

[mholt]

Can you be more specific?

What are the specific instructions I can follow to reproduce the behavior?

[DaoYangM]

@mholt https://github.com/v2ray/discussion/issues/296 This issue has been solved in Caddy1 but it seems appear in Caddy2. The contents are Chinese you need to translate

[mholt]

The Google translation is not exactly helpful. Can you provide an English translation of the steps to reproduce the bug? Otherwise I do not think there is anything to fix, it all seems to be working fine here, because I cannot reproduce it.

[DaoYangM]

@mholt We broke through China's Internet blockade by configuring v2ray. The configuration process is a bit complicated. The calling process looks like this. browser Input www.google.com -> v2ray client -> caddy -> v2ray server then back to browser. The issue is talking about that does caddy support with h2c reverse proxy ?

[DaoYangM]

@mholt Please forgive me for expressing ambiguity, because I don't know much about the details. I may close this issue, and waiting people who have specific details submit an issue to you. Thanks for the reply.

[mholt]

Thanks for trying to explain. I know it's complicated -- that's why we need to be specific about what is "not working". I don't blame you, really, I'm just trying to narrow it down.

The issue is talking about that does caddy support with h2c reverse proxy ?

Ah, now we're getting somewhere. This is correct. Caddy 2 (and Go's standard library) does not support HTTP/2 over plaintext. (This is the same as Caddy 1.)

We might be able to add support for it with this: https://godoc.org/golang.org/x/net/http2/h2c -- but I've never used this package.

Would potentially be willing to review a PR for this, if you need it -- but I just need to decide if I want to help proliferate the spread of unencrypted transmissions across the WAN. It seems to rub against Caddy's values.

[mholt]

@DaoYangM I'm implementing h2c into the v2 reverse proxy over in https://github.com/caddyserver/caddy/issues/3218 -- the h2c branch. If you could try it out and join the discussion there, that'd be helpful!

[choicky]

@DaoYangM I'm implementing h2c into the v2 reverse proxy over in #3218 -- the h2c branch. If you could try it out and join the discussion there, that'd be helpful!

I'm upgrading from caddy v1 to caddy v2.

The following Caddyfile configuration of h2 backend proxy works fine in caddy v1.

    ## h2 backend proxy, using cert of MyDomain.com
    proxy /Myh2Path https://localhost:35486 {
        insecure_skip_verify
        header_upstream X-Forwarded-Proto "https"
        header_upstream Host "MyDomain.com"
    }

However, neither caddy v2.0 nor caddy v2.1.0-beta.1 works with the following Caddyfile:

    reverse_proxy /Myh2Path {
        to https://localhost:35486
        header_up X-Forwarded-Proto "https"
        header_up Host "MyDomain.com"
        transport http {
            tls
            tls_insecure_skip_verify
            versions 2
        }
    }

The Caddyfile is converted into json:

    {
        "handle":[
            {
                "handler":"reverse_proxy",
                "headers":{
                    "request":{
                        "set":{
                            "Host":[
                                "MyDomain.com"],
                            "X-Forwarded-Proto":[
                                "https"]
                        }
                    }
                },
                "transport":{
                    "protocol":"http",
                    "tls":{
                        "insecure_skip_verify":true
                    },
                    "versions":[
                        "2"]
                },
                "upstreams":[
                    {
                        "dial":"localhost:35486"
                    }]
            }],
        "match":[
            {
                "path":[
                    "/Myh2Path "]
            }]
    },

[francislavoie]

@choicky You're encountering a completely unrelated problem. This issue has to do with h2c, you're not using h2c.

In Caddy v2, matchers are exact-match. That means that your /Myh2Path path matcher will only match /Myh2Path and not /Myh2Path/foo. Append a * to tell Caddy to match any subpath.

You can simplify your config to this (the rest is unnecessary; setting the Host is probably unnecessary as well unless your site is specifically different than what you're sending to your proxy, Caddy v2 passes through all headers transparently already):

    reverse_proxy /Myh2Path* https://localhost:35486 {
        header_up Host "MyDomain.com"
        transport http {
            tls_insecure_skip_verify
        }
    }

For next time, please ask your usage questions on the Caddy community forums. We prefer to keep the GitHub issue board for bugs and feature requests. Don't forget to fill out the thread template so we can help you!

[choicky]

@choicky You're encountering a completely unrelated problem. This issue has to do with h2c, you're not using h2c.

In Caddy v2, matchers are exact-match. That means that your /Myh2Path path matcher will only match /Myh2Path and not /Myh2Path/foo. Append a * to tell Caddy to match any subpath.

You can simplify your config to this (the rest is unnecessary; setting the Host is probably unnecessary as well unless your site is specifically different than what you're sending to your proxy, Caddy v2 passes through all headers transparently already):

    reverse_proxy /Myh2Path* https://localhost:35486 {
        header_up Host "MyDomain.com"
        transport http {
            tls_insecure_skip_verify
        }
    }

For next time, please ask your usage questions on the Caddy community forums. We prefer to keep the GitHub issue board for bugs and feature requests. Don't forget to fill out the thread template so we can help you!

Thanks for your reply.

The problems I encountered may be a completely unrelated problem, technically. However, as an ordinary user, I think the problem I encountered is related to this topic. The configuration file that works in caddy v1 does not work in caddy v2.

[francislavoie]

No, it isn't related. h2c is "HTTP/2 over cleartext TCP". This is not the same thing as you encountered.

I'm locking this thread, it's resolved and you started an unrelated discussion.