search

caddyserver / caddy

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
https://caddyserver.com
Apache License 2.0
58.11k stars 4.03k forks source link

TLS handshake error from 127.0.0.1: EOF, then crash #3590

Closed Don-Swanson closed 4 years ago

Don-Swanson commented 4 years ago

Hello, I'm having an issue lately where I'll be notified by my monitoring service that my sites are no longer reachable. When I try to visit I get a Server 521 error. When I look at the logs, there is a string of TLS handshake error from 127.0.0.1:50836: EOF errors right before the server crashes. If I restart the container, it comes back up, but it crashes anywhere between 5 and 15 times per week. Any Ideas what's going on? Anything else I can do to troubleshoot?

{"level":"info","ts":1595260091.5539384,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1595260091.5692425,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1595260091.5700438,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2020/07/20 15:48:11 [INFO][cache:0xc0005c8fc0] Started certificate maintenance routine
{"level":"info","ts":1595260091.5700884,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1595260091.5791159,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1595260091.5794544,"logger":"http","msg":"enabling automatic TLS certificate management","domains":[***LIST OF DOMAINS SERVED***]}
{"level":"info","ts":1595260091.806902,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1595260091.8069775,"msg":"serving initial configuration"}
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50856: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50854: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50844: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50852: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50846: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50842: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50840: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50850: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50838: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50836: EOF
francislavoie commented 4 years ago

Thanks for opening an issue! We'll look into this.

It's not immediately clear to us what is going on, so we'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1:

I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

  1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

  2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

    • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
    • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
    • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

  3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

    • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
    • Do not run Caddy inside containers.
    • Run Caddy manually in your terminal; do not use systemd or other init systems.
    • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
    • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
    • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.
Don-Swanson commented 4 years ago

Thanks for the quick Reply:

1. Environment

1a. Operating system and version

Fedora 32 running podman 2.0.2

1b. Caddy version (run caddy version or paste commit SHA)

2.1.1 (Running In a container)

2. Description

2a. What happens (briefly explain what is wrong)

Caddy crashes after a series of TLS handshake error from 127.0.0.1:50842: EOF errors

2b. Why it's a bug (if it's not obvious)

Because caddy shouldn't be crashing

2c. Log output

{"level":"info","ts":1595260091.5539384,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1595260091.5692425,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1595260091.5700438,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2020/07/20 15:48:11 [INFO][cache:0xc0005c8fc0] Started certificate maintenance routine
{"level":"info","ts":1595260091.5700884,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1595260091.5791159,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1595260091.5794544,"logger":"http","msg":"enabling automatic TLS certificate management","domains":[***LIST OF DOMAINS SERVED***]}
{"level":"info","ts":1595260091.806902,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1595260091.8069775,"msg":"serving initial configuration"}
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50856: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50854: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50844: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50852: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50846: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50842: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50840: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50850: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50838: EOF
2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50836: EOF

2d. Workaround(s)

None, have to restart container on crash.

2e. Relevant links

None

3. Tutorial (minimal steps to reproduce the bug)

Create the container to use Cloudflare DNS Auth dockerfile:

FROM caddy:builder AS builder
RUN caddy-builder \
    github.com/caddy-dns/cloudflare
FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Run docker build --tag caddy:cf -f Dockerfile

Create Caddyfile:

{
        email support@mydomain.com
}

data.mydomain.com {

    header {
        #Custom Header
        Custom-Header "‘; DROP TABLE servertypes; —"
        X-Powered-By "PFM and Unicorns"
        X-HTTP-Status "418 I`m a teapot"
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
    }
    tls {
        dns cloudflare CLOUDFLAREKEY
    }
    # Proxy the Root directory
    reverse_proxy 192.168.2.252:3003 {
    }
}

######################################
gitlab.mydomain.com {

    header {
        #Custom Header
        Custom-Header "‘; DROP TABLE servertypes; —"
        X-Powered-By "PFM and Unicorns"
        X-HTTP-Status "418 I`m a teapot"
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
    }
    tls {
        dns cloudflare CLOUDFLAREKEY
    }
   # Proxy the Root directory
    reverse_proxy https://192.168.2.252:3021 {
        transport http {
            tls
            tls_insecure_skip_verify
            }
    }
}

######################################
registry.mydomain.com {

    header {
        #Custom Header
        Custom-Header "‘; DROP TABLE servertypes; —"
        X-Powered-By "PFM and Unicorns"
        X-HTTP-Status "418 I`m a teapot"
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
    }
    tls {
        dns cloudflare CLOUDFLAREKEY
    }
   # Proxy the Root directory
    reverse_proxy http://192.168.2.252:5005 {
#        transport http {
#            tls
#            tls_insecure_skip_verify
#            }
    }
}
######################################
smoke.mydomain.com {

    header {
        #Custom Header
        Custom-Header "‘; DROP TABLE servertypes; —"
        X-Powered-By "PFM and Unicorns"
        X-HTTP-Status "418 I`m a teapot"
        # Enable HTTP Strict Transport Security (HSTS)
        Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
    }
    tls {
        dns cloudflare CLOUDFLAREKEY
    }
    # Proxy the Root directory
    reverse_proxy 192.168.2.252:3002 {
    }
}

Run the Caddy container: docker run -d --name caddy -p 3000:80 -p 3100:443 -v caddy:/data -v /storage/caddy/Caddyfile:/etc/caddy/Caddyfile -v caddy_config:/config caddy:cf

francislavoie commented 4 years ago

HTTP status 521 comes from Cloudflare. What is Caddy actually returning in those situations?

When you say "crashes", there's no clear evidence of that. I'm not seeing a panic from Caddy. Can you clarify what you mean? What evidence of "crashing" is there?

mholt commented 4 years ago

Thanks, but I have a few more questions. The log output -- is that all there is? A "crash" usually involves a stack trace. Can you define exactly what you mean by "crash"? Is the OS killing the process? What is the full log output?

You say "run the container" -- but then what? Is that all? Does it crash without any requests to it? Please provide enough detail so we can reproduce it.

Don-Swanson commented 4 years ago

@francislavoie I'll have to try and check next time it crashes for you.

@francislavoie and @mholt Yes, I know there is no evidence of it crashing in the logs, but the only way to resolve it is to restart the container. If you have a way for me to enable debugging I can get you more in-depth logs. I do not see an option for debug for caddyfile nor for a command line option to pass when starting the container.

I have no idea what is causing it to crash, those handshake errors are the only thing I see in the log and there are a whole bunch of them in a row right at the same time that my sites stop being served. It doesn't crash instantly, I know that requests are going to it all the time because at a minimum I have uptime monitors checking every few minutes.

I guess the next step is how I can enable debug logs so I can see what's going on and get you more of the details you are requesting.

mholt commented 4 years ago

A good start would be enabling access logs: log

And debug logs: debug in your Caddyfile global options at the top.

2020/07/20 16:02:36 http: TLS handshake error from 127.0.0.1:50836: EOF just means that the client is attempting to connect to a TLS socket but it isn't sending any data. Who/what are your clients?

Don-Swanson commented 4 years ago

I get this when I add log to the caddyfile: "level":"info","ts":1595268117.4018092,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"} run: adapting config using caddyfile: /etc/caddy/Caddyfile:4: unrecognized global option: log

As for clients... just web browsers (and whatever the uptime monitors use).

mholt commented 4 years ago

@TheEagle13 log is not a global option, it is a site directive: https://caddyserver.com/docs/caddyfile/directives/log

Don-Swanson commented 4 years ago

OK, I started it with the debug option, and then I just started refreshing everything that I have hosted, and it appears to be happening after I start refreshing and loading a lot of things from my octoprint server. Logs are below. When it does "crash" and I try to connect directly I just get "Firefox can’t establish a connection to the server at 192.168.1.252:3100."

{"level":"debug","ts":1595268623.9994981,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/static/img/mask-theme.svg","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57472","host":"print.mydomain.com","headers":{"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cache-Control":["no-cache"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeDg._lK1d19BSFRwa9rwC6QHvUMqbiE; __utmc=52526903"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"Dnt":["1"],"Pragma":["no-cache"],"Cf-Request-Id":["040f0485c00000e0512300b200000001"],"Cdn-Loop":["cloudflare"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ea382ce6be051-DFW"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept":["image/webp,*/*"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.112579228,"headers":{"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeDw.uLdqYeVgoj4qA_EcWJILZGKIass; HttpOnly; Path=/"],"Etag":["\"1591291575.53-2046-1243094993\""],"Cache-Control":["public, max-age=43200"],"Date":["Mon, 20 Jul 2020 18:10:23 GMT"],"Content-Type":["image/svg+xml; charset=utf-8"],"X-Frame-Options":["sameorigin"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"X-Content-Type-Options":["nosniff"],"Expires":["Tue, 21 Jul 2020 06:10:23 GMT"],"Last-Modified":["Thu, 04 Jun 2020 17:26:15 GMT"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Length":["2046"],"Accept-Ranges":["bytes"]},"status":200}
{"level":"debug","ts":1595268625.0443747,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/sockjs/info?t=1595268624888","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57474","host":"print.mydomain.com","headers":{"Cf-Ray":["5b5ea389ed17e051-DFW"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeDw.uLdqYeVgoj4qA_EcWJILZGKIass; __utmc=52526903"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.5"],"Cf-Request-Id":["040f048a330000e05123060200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.038441981,"headers":{"X-Content-Type-Options":["nosniff"],"Access-Control-Allow-Credentials":["true"],"Date":["Mon, 20 Jul 2020 18:10:25 GMT"],"X-Frame-Options":["sameorigin"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Cache-Control":["no-store, no-cache, must-revalidate, max-age=0"],"Etag":["\"94c35dc67097fab4e29819d47842fdd39280bbfe\""],"Access-Control-Allow-Origin":["*"],"Content-Type":["application/json; charset=UTF-8"],"Content-Length":["78"]},"status":200}
{"level":"debug","ts":1595268625.2927337,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/sockjs/839/rkzt4lvd/websocket","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57476","host":"print.mydomain.com","headers":{"Cf-Ray":["5b5ea38b58fc9adc-DFW"],"Accept-Encoding":["gzip"],"Accept":["*/*"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Cf-Request-Id":["040f048b1300009adc29ab5200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ipcountry":["US"],"Upgrade":["websocket"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"Sec-Websocket-Key":["AeTWa1J/wKQXEpFjcOXxgg=="],"Dnt":["1"],"Connection":["Upgrade"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Websocket-Version":["13"],"Origin":["https://print.mydomain.com"],"Sec-Websocket-Extensions":["permessage-deflate"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeDw.uLdqYeVgoj4qA_EcWJILZGKIass; __utmc=52526903"],"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.044259957,"headers":{"Upgrade":["websocket"],"Sec-Websocket-Extensions":["permessage-deflate"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Sec-Websocket-Accept":["fhcalfFsUk2zurbZTQ3karGW9G4="],"X-Content-Type-Options":["nosniff"],"Connection":["Upgrade"],"Date":["Mon, 20 Jul 2020 18:10:25 GMT"],"X-Frame-Options":["sameorigin"]},"status":101}
{"level":"debug","ts":1595268626.7563438,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"POST","uri":"/api/login","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57482","host":"print.mydomain.com","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Cache-Control":["no-cache"],"X-Requested-With":["XMLHttpRequest"],"Cf-Request-Id":["040f048fbf0000e051230b4200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["5b5ea392c808e051-DFW"],"Dnt":["1"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeDw.uLdqYeVgoj4qA_EcWJILZGKIass; __utmc=52526903"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept-Language":["en-US,en;q=0.5"],"Origin":["https://print.mydomain.com"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["16"],"X-Forwarded-Proto":["https"],"Content-Type":["application/json; charset=UTF-8"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.330806818,"headers":{"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeEg.nyOOnZP8h5UscWh7FDSCqLfyu2M; HttpOnly; Path=/"],"Pragma":["no-cache"],"X-Frame-Options":["sameorigin"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["application/json"],"Content-Length":["838"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Expires":["-1"],"Cache-Control":["pre-check=0, no-cache, no-store, must-revalidate, post-check=0, max-age=0"]},"status":200}
{"level":"debug","ts":1595268628.3711638,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/settings","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57486","host":"print.mydomain.com","headers":{"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept":["application/json, text/javascript, */*; q=0.01"],"X-Requested-With":["XMLHttpRequest"],"Dnt":["1"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"If-Modified-Since":["Mon, 20 Jul 2020 15:00:00 GMT"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip"],"Cf-Request-Id":["040f04919a0000e051230d4200000001"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ray":["5b5ea395cd59e051-DFW"],"If-None-Match":["W/\"21166d444a01dcb53a3be702232734ea4e27a189\""],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeEg.nyOOnZP8h5UscWh7FDSCqLfyu2M; __utmc=52526903"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":1.468491173,"headers":{"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; HttpOnly; Path=/"],"Cache-Control":["max-age=0"],"Content-Length":["12880"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Etag":["\"21166d444a01dcb53a3be702232734ea4e27a189\""],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["application/json"],"X-Frame-Options":["sameorigin"],"X-Content-Type-Options":["nosniff"],"Last-Modified":["Mon, 20 Jul 2020 15:00:00 GMT"]},"status":200}
{"level":"debug","ts":1595268629.0303578,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/access/groups","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57486","host":"print.mydomain.com","headers":{"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"X-Requested-With":["XMLHttpRequest"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Cf-Ray":["5b5ea3a27ea4e051-DFW"],"X-Forwarded-Proto":["https"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Cf-Request-Id":["040f04998d0000e0512314c200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.147225698,"headers":{"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFQ.Bo8DWLGGs4o-REBMD-LASQAQdjU; HttpOnly; Path=/"],"Cache-Control":["max-age=0"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["application/json"],"X-Frame-Options":["sameorigin"],"Content-Length":["2576"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"X-Content-Type-Options":["nosniff"]},"status":200}
{"level":"debug","ts":1595268629.8738618,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/printerprofiles","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57482","host":"print.mydomain.com","headers":{"X-Requested-With":["XMLHttpRequest"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept":["application/json, text/javascript, */*; q=0.01"],"If-None-Match":["W/\"b00346d2a168bb862114c3257b868775ee7c6a64\""],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ea3a28ebce051-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cf-Ipcountry":["US"],"Cf-Request-Id":["040f0499920000e0512314e200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"If-Modified-Since":["Mon, 20 Jul 2020 14:57:55 GMT"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.981072502,"headers":{"X-Content-Type-Options":["nosniff"],"Last-Modified":["Mon, 20 Jul 2020 14:57:55 GMT"],"Etag":["\"b00346d2a168bb862114c3257b868775ee7c6a64\""],"X-Clacks-Overhead":["GNU Terry Pratchett"],"X-Frame-Options":["sameorigin"],"Content-Length":["672"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFQ.Bo8DWLGGs4o-REBMD-LASQAQdjU; HttpOnly; Path=/"],"Cache-Control":["max-age=0"],"Content-Type":["application/json"]},"status":200}
{"level":"debug","ts":1595268630.0001576,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/system/commands","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57452","host":"print.mydomain.com","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"Dnt":["1"],"Cf-Ipcountry":["US"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cf-Request-Id":["040f0499920000e0512314f200000001"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["5b5ea3a28ebfe051-DFW"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"X-Requested-With":["XMLHttpRequest"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":1.106610926,"headers":{"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFQ.Bo8DWLGGs4o-REBMD-LASQAQdjU; HttpOnly; Path=/"],"Cache-Control":["max-age=0"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["application/json"],"X-Frame-Options":["sameorigin"],"Content-Length":["2200"],"X-Robots-Tag":["noindex, nofollow, noimageindex"]},"status":200}
{"level":"debug","ts":1595268632.3512998,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57286","host":"print.mydomain.com","headers":{"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ea29f1e7ee051-DFW"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLkME7VdhR1s48TBP0HIlOQi23TGGLtRlVpLv_vsTr3s9rzwe-C5oOLrWPUaBcNxrFaoaEoUXNDDBwqQ7NJJ5plmIvW5eCMsTl2Y5SRNfeKCWYgTT5myBWo7ZrRRIvKAphMI2y7a53nWKao7oOGJ09aFDhwQavGZ5WLXQg6zEnJmMZBliwBJZ05rLHPpKpH4jH5ilidLlk7MgMfoq5HCatVFjhQJ5kIbOdcnmOEZXVeoKavD0Aznx_dx0MVw_qlQcBitvWvu3ql4O96osa-OxT95X_V98334w5DrhGsvIhHG29DbvGyfop0bhR7F_n5D9muMrr9Jkmox.XxXd6g.b5Dsuk0rbsU8emogA-1lIGKKw28; __utmc=52526903"],"Cf-Connecting-Ip":["136.50.25.2"],"Dnt":["1"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.5"],"Cf-Request-Id":["040f03f7710000e05123345200000001"],"Pragma":["no-cache"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":44.903966801,"headers":{"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLkME7VdhR1s48TBP0HIlOQi23TGGLtRlVpLv_vsTr3s9rzwe-C5oOLrWPUaBcNxrFaoaEoUXNDDBwqQ7NJJ5plmIvW5eCMsTl2Y5SRNfeKCWYgTT5myBWo7ZrRRIvKAphMI2y7a53nWKao7oOGJ09aFDhwQavGZ5WLXQg6zEnJmMZBliwBJZ05rLHPpKpH4jH5ilidLlk7MgMfoq5HCatVFjhQJ5kIbOdcnmOEZXVeoKavD0Aznx_dx0MVw_qlQcBitvWvu3ql4O96osa-OxT95X_V98334w5DrhGsvIhHG29DbvGyfop0bhR7F_n5D9muMrr9Jkmox.XxXeGA.tDPEl8hDJJ6G0KUtwM6zL_5w3dw; HttpOnly; Path=/"],"Last-Modified":["Thu, 16 Jul 2020 14:51:05 GMT"],"Etag":["\"37487b176a95079cc4785e15960b97c2f8db0a1f\""],"Content-Type":["text/html; charset=utf-8"],"X-Frame-Options":["sameorigin"],"Content-Length":["543918"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Cache-Control":["no-cache, must-revalidate"],"X-Clacks-Overhead":["GNU Terry Pratchett"]},"status":200}
{"level":"debug","ts":1595268632.4128785,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/sockjs/info?t=1595268610729","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57446","host":"print.mydomain.com","headers":{"Cdn-Loop":["cloudflare"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.5"],"Cf-Request-Id":["040f0452d00000e051230d7200000001"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ea3314904e051-DFW"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept":["*/*"],"Pragma":["no-cache"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLkME7VdhR1s48TBP0HIlOQi23TGGLtRlVpLv_vsTr3s9rzwe-C5oOLrWPUaBcNxrFaoaEoUXNDDBwqQ7NJJ5plmIvW5eCMsTl2Y5SRNfeKCWYgTT5myBWo7ZrRRIvKAphMI2y7a53nWKao7oOGJ09aFDhwQavGZ5WLXQg6zEnJmMZBliwBJZ05rLHPpKpH4jH5ilidLlk7MgMfoq5HCatVFjhQJ5kIbOdcnmOEZXVeoKavD0Aznx_dx0MVw_qlQcBitvWvu3ql4O96osa-OxT95X_V98334w5DrhGsvIhHG29DbvGyfop0bhR7F_n5D9muMrr9Jkmox.XxXd6g.b5Dsuk0rbsU8emogA-1lIGKKw28; __utmc=52526903"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Dnt":["1"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":21.581714529,"headers":{"X-Content-Type-Options":["nosniff"],"Access-Control-Allow-Credentials":["true"],"Date":["Mon, 20 Jul 2020 18:10:32 GMT"],"Access-Control-Allow-Origin":["*"],"X-Frame-Options":["sameorigin"],"Content-Length":["79"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Cache-Control":["no-store, no-cache, must-revalidate, max-age=0"],"Etag":["\"ff8b91e6c3fc256c2ad1d654102c839f41f02319\""],"Content-Type":["application/json; charset=UTF-8"]},"status":200}
{"level":"debug","ts":1595268632.455965,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/sockjs/info?t=1595268618747","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57454","host":"print.mydomain.com","headers":{"Accept-Encoding":["gzip"],"Pragma":["no-cache"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Cf-Ray":["5b5ea3636b94e051-DFW"],"X-Forwarded-Proto":["https"],"Accept":["*/*"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLkME7VdhR1s48TBP0HIlOQi23TGGLtRlVpLv_vsTr3s9rzwe-C5oOLrWPUaBcNxrFaoaEoUXNDDBwqQ7NJJ5plmIvW5eCMsTl2Y5SRNfeKCWYgTT5myBWo7ZrRRIvKAphMI2y7a53nWKao7oOGJ09aFDhwQavGZ5WLXQg6zEnJmMZBliwBJZ05rLHPpKpH4jH5ilidLlk7MgMfoq5HCatVFjhQJ5kIbOdcnmOEZXVeoKavD0Aznx_dx0MVw_qlQcBitvWvu3ql4O96osa-OxT95X_V98334w5DrhGsvIhHG29DbvGyfop0bhR7F_n5D9muMrr9Jkmox.XxXd6g.b5Dsuk0rbsU8emogA-1lIGKKw28; __utmc=52526903"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Cf-Connecting-Ip":["136.50.25.2"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Cache-Control":["no-cache"],"Cf-Request-Id":["040f04721e0000e051232c3200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":13.615177214,"headers":{"Access-Control-Allow-Origin":["*"],"Content-Length":["79"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"X-Content-Type-Options":["nosniff"],"Cache-Control":["no-store, no-cache, must-revalidate, max-age=0"],"Etag":["\"e31bff8f833e39a3d257001ac778e36f0e75265c\""],"Access-Control-Allow-Credentials":["true"],"Date":["Mon, 20 Jul 2020 18:10:32 GMT"],"Content-Type":["application/json; charset=UTF-8"],"X-Frame-Options":["sameorigin"]},"status":200}
{"level":"debug","ts":1595268634.8463678,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/plugin/announcements/channels","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57432","host":"print.mydomain.com","headers":{"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ray":["5b5ea3a28ec3e051-DFW"],"Accept-Language":["en-US,en;q=0.5"],"X-Requested-With":["XMLHttpRequest"],"Dnt":["1"],"If-None-Match":["W/\"22ba795c7c1d3e02f5258b5da8e6429d20565736\""],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json, text/javascript, */*; q=0.01"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Cf-Ipcountry":["US"],"Cf-Request-Id":["040f0499920000e05123152200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":5.950859848,"headers":{"X-Content-Type-Options":["nosniff"],"Cache-Control":["max-age=0"],"Content-Type":["application/json"],"Etag":["\"22ba795c7c1d3e02f5258b5da8e6429d20565736\""],"X-Clacks-Overhead":["GNU Terry Pratchett"],"X-Frame-Options":["sameorigin"],"Content-Length":["30076"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeGg.pp3ljCRWQCioelYEFe9mxjXtyVI; HttpOnly; Path=/"]},"status":200}
{"level":"debug","ts":1595268635.452597,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/plugin/PrintTimeGenius/print_history","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57492","host":"print.mydomain.com","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cf-Connecting-Ip":["136.50.25.2"],"X-Forwarded-Proto":["https"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Dnt":["1"],"Cf-Ray":["5b5ea3a23e03e051-DFW"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept-Language":["en-US,en;q=0.5"],"X-Requested-With":["XMLHttpRequest"],"Cf-Request-Id":["040f0499600000e05123146200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":6.554415272,"headers":{"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeGw.qwYGGGeXx5ZOboM0hsc8tIQHQhI; HttpOnly; Path=/"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["text/html; charset=utf-8"],"X-Frame-Options":["sameorigin"],"Content-Length":["2524"],"X-Robots-Tag":["noindex, nofollow, noimageindex"]},"status":200}
{"level":"debug","ts":1595268635.5513716,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/plugin/pi_support","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57494","host":"print.mydomain.com","headers":{"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cf-Ray":["5b5ea3a28ec0e051-DFW"],"Accept-Language":["en-US,en;q=0.5"],"X-Requested-With":["XMLHttpRequest"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Cf-Request-Id":["040f0499920000e05123150200000001"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Dnt":["1"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":6.604372283,"headers":{"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["application/json"],"X-Frame-Options":["sameorigin"],"Content-Length":["243"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeGw.qwYGGGeXx5ZOboM0hsc8tIQHQhI; HttpOnly; Path=/"],"Cache-Control":["max-age=0"]},"status":200}
{"level":"debug","ts":1595268635.6669502,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"GET","uri":"/api/timelapse?unrendered=true","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57496","host":"print.mydomain.com","headers":{"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Accept-Language":["en-US,en;q=0.5"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["5b5ea3a28ec1e051-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Requested-With":["XMLHttpRequest"],"If-None-Match":["W/\"7d643bef7fca0ebb5a7fc555e234d1f335f26b5c\""],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Dnt":["1"],"If-Modified-Since":["Mon, 20 Jul 2020 14:57:55 GMT"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeFA.5JJ248bQZI7m7w1-wWWTfc_IgTg; __utmc=52526903"],"Cf-Request-Id":["040f0499920000e05123151200000001"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":6.717381396,"headers":{"Content-Type":["application/json"],"Content-Length":["69"],"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeGw.qwYGGGeXx5ZOboM0hsc8tIQHQhI; HttpOnly; Path=/"],"Last-Modified":["Mon, 20 Jul 2020 14:57:55 GMT"],"Etag":["\"7d643bef7fca0ebb5a7fc555e234d1f335f26b5c\""],"Cache-Control":["max-age=0"],"X-Robots-Tag":["noindex, nofollow, noimageindex"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"X-Frame-Options":["sameorigin"]},"status":200}
{"level":"debug","ts":1595268636.0928373,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.7.0.5:80","request":{"method":"POST","uri":"/plugin/PrintTimeGenius/print_history","proto":"HTTP/1.1","remote_addr":"127.0.0.1:57452","host":"print.mydomain.com","headers":{"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"],"Cdn-Loop":["cloudflare"],"Accept-Language":["en-US,en;q=0.5"],"Cache-Control":["no-cache"],"Cf-Request-Id":["040f04b37d0000e05123334200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Requested-With":["XMLHttpRequest"],"Origin":["https://print.mydomain.com"],"Dnt":["1"],"Cf-Ray":["5b5ea3cbfc39e051-DFW"],"Content-Length":["2391"],"Accept":["application/json, text/javascript, */*; q=0.01"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Type":["application/json; charset=UTF-8"],"Cookie":["__cfduid=d0fc083270c9ce8d7bea862fcd7c012481594917945; __utma=52526903.577014220.1594917941.1595094745.1595259147.3; __utmz=52526903.1594917941.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=52526903.|1=logged-in=no=1; session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeGw.qwYGGGeXx5ZOboM0hsc8tIQHQhI; __utmc=52526903; cf_ob_info=525:5b5ea3a28ec5e051:DFW; cf_use_ob=0"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"print.mydomain.com"}},"duration":0.567583625,"headers":{"X-Content-Type-Options":["nosniff"],"Set-Cookie":["session_P443=.eJx1j01rgzAcxr_KyLmMGqNthR3mUsXBP0HIlOQi23SNMXajKlpLv_vsTr3s9rzwe-C5oOLrVHUaBf1pqFaoqEsUXNDDBwqQbNNJ5plmIvW4eMMsTl2Y5STNYeSCWYgTokzZALUtM9oosSdA0wmEbRbt8TxrFdUt0OeR08aFFtYg1OIzy0XYQA6zEnJmMeBlCwNOZ04Pjsylq0TiMfrpsDxZsnRiBgijr0YKq1W7X0uROFxoI-fDCDM8oesK1WV17Ov-_Pg-9Lrozz8VCo6DtXfN3TsV74YbNXTVqfgn76quq7-PfxgK3S32dn6EN5FL6DrcRn70QkgYEs_3ww1G119KJWoz.XxXeHA.npiW-WJh0-49QxqND8H1y2BUNuc; HttpOnly; Path=/"],"Cache-Control":["no-cache"],"X-Clacks-Overhead":["GNU Terry Pratchett"],"Content-Type":["text/html; charset=utf-8"],"X-Frame-Options":["sameorigin"],"Content-Length":["0"],"X-Robots-Tag":["noindex, nofollow, noimageindex"]},"status":200}
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57506: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57502: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57504: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57498: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57508: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57500: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57516: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57514: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57510: EOF
2020/07/20 18:12:06 http: TLS handshake error from 127.0.0.1:57512: EOF
mholt commented 4 years ago

Can you reproduce it without Docker? I'm suspecting something about the Docker setup at this point, because there's nothing in Caddy indicating a problem: the only issue is that bytes stop being deliverable to Caddy, which is probably something outside of Caddy causing that.

Don-Swanson commented 4 years ago

well, bytes are still going to caddy. things stopped loading after the last debug log, then I double-checked the logs before posting a minute later and all those handshake error logs appeared. So those actually appear about 30 seconds to a minute after I stop being able to load pages.

mholt commented 4 years ago

well, bytes are still going to caddy.

Can you prove that? The logs say EOF, meaning that the streams are being terminated.

Please try without Docker and let's see what happens. This is something we've been requesting since the beginning to help narrow down the troubleshooting process more quickly:

  • Do not run Caddy inside containers.
Don-Swanson commented 4 years ago

Well, It won't be until the weekend before I can have time to build up a VM, install caddy and migrate to test.

As for your don't run caddy inside containers comment, if you don't want us to run it in containers, then why do you release official Docker images?

mholt commented 4 years ago

Well, It won't be until the weekend before I can have time to build up a VM, install caddy and migrate to test.

That's probably fine, you're the only one experiencing the issue so far, so take your time.

As for your don't run caddy inside containers comment, if you don't want us to run it in containers, then why do you release official Docker images?

You're missing the point. 🙂 Our bug report template has those instructions so that we can pinpoint the cause of the bug faster, not because "we don't want people to run it in containers," obviously. I even explained this in my comment above:

to help narrow down the troubleshooting process more quickly

francislavoie commented 4 years ago

When it "crashes", did the container die, or is it still running but hung? If it hung, get a shell into the container with docker exec or docker-compose exec, then run curl localhost:2019/config/? (You may need to install curl with apk add curl). This should hit Caddy's admin API. Do you get a JSON response?

Don-Swanson commented 4 years ago

I just did some additional tests (I had previously disabled my GitLab runner from calling to check in because it was creating so much chatter Checking in every few seconds). I re-enabled it, ran the tests again. The server stopped serving requests to any of my sites, but it looks like the container continues to serve the GitLab runner requests. (The main difference here is that these are POSTs instead of GETs)

@francislavoie The container stays up and running. When I exec into the container (install curl) then do the curl, yes I get a response that looks like the JSON config file.

(EDIT forgot to add the logs)

2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60168: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60156: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60158: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60160: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60152: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60164: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60154: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60162: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60166: EOF
2020/07/20 19:04:04 http: TLS handshake error from 127.0.0.1:60150: EOF
{"level":"debug","ts":1595271844.3878143,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58490","host":"gitlab.mydomain.com","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f35aa1b0000e049d9add200000001"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"X-Forwarded-Proto":["https"],"Content-Type":["application/json"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["5b5ef223593fe049-DFW"],"Content-Length":["518"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002836412,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:04 GMT"]},"status":204}
{"level":"debug","ts":1595271845.8919072,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58460","host":"gitlab.mydomain.com","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Content-Type":["application/json"],"Cf-Ipcountry":["US"],"Cf-Request-Id":["040f35aff700000f2214346200000001"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ef22cba940f22-DFW"],"Accept":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Content-Length":["518"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002999771,"headers":{"Date":["Mon, 20 Jul 2020 19:04:05 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"]},"status":204}
{"level":"debug","ts":1595271847.3886151,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58468","host":"gitlab.mydomain.com","headers":{"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Type":["application/json"],"Cf-Request-Id":["040f35b5d30000e049d9bc7200000001"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Content-Length":["518"],"Cf-Ray":["5b5ef2361ae1e049-DFW"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002969404,"headers":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:07 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"]},"status":204}
{"level":"debug","ts":1595271848.8914037,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58454","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Accept":["application/json"],"Accept-Encoding":["gzip"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["5b5ef23f8bf40f22-DFW"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Content-Type":["application/json"],"Cf-Request-Id":["040f35bbb100000f221406e200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.003208765,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:08 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271850.3877428,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58472","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Content-Type":["application/json"],"Cf-Request-Id":["040f35c18c0000e049d9889200000001"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef248e914e049-DFW"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"],"Content-Length":["518"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002133328,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:10 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271851.8920188,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58440","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cf-Ray":["5b5ef2524f1f0f22-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Content-Type":["application/json"],"Cf-Request-Id":["040f35c76900000f2214177200000001"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Accept":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002884639,"headers":{"Date":["Mon, 20 Jul 2020 19:04:11 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"]},"status":204}
{"level":"debug","ts":1595271853.3898408,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58472","host":"gitlab.mydomain.com","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Content-Type":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Cf-Ipcountry":["US"],"Cf-Request-Id":["040f35cd450000e049d9952200000001"],"X-Forwarded-Proto":["https"],"Cf-Ray":["5b5ef25baea8e049-DFW"],"Content-Length":["518"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.003059429,"headers":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:13 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"]},"status":204}
{"level":"debug","ts":1595271854.8951418,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58444","host":"gitlab.mydomain.com","headers":{"Cf-Connecting-Ip":["136.50.25.2"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ef2650e640f22-DFW"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Content-Type":["application/json"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Accept-Encoding":["gzip"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f35d32200000f221429d200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.003101079,"headers":{"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:14 GMT"],"Gitlab-Ci-Builds-Polling":["yes"]},"status":204}
{"level":"debug","ts":1595271856.3989706,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58472","host":"gitlab.mydomain.com","headers":{"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Cf-Request-Id":["040f35d8ff0000e049d9a0d200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef26e6c99e049-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ipcountry":["US"],"Content-Length":["518"],"Content-Type":["application/json"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002834168,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:16 GMT"]},"status":204}
{"level":"debug","ts":1595271857.8911421,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58440","host":"gitlab.mydomain.com","headers":{"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Content-Type":["application/json"],"Cf-Request-Id":["040f35ded900000f22143cc200000001"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ef277cd3b0f22-DFW"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.004279795,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:17 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271859.3920038,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58498","host":"gitlab.mydomain.com","headers":{"X-Forwarded-Proto":["https"],"Cf-Connecting-Ip":["136.50.25.2"],"Content-Length":["518"],"Cf-Request-Id":["040f35e4b70000e049d9aa1200000001"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef2812aa3e049-DFW"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Content-Type":["application/json"],"Cf-Ipcountry":["US"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.00288672,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:19 GMT"]},"status":204}
{"level":"debug","ts":1595271860.8903205,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58484","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Content-Length":["518"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ef28a8b180f22-DFW"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Content-Type":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"Cdn-Loop":["cloudflare"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cf-Request-Id":["040f35ea9200000f2214110200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.003016588,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:20 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271862.3923995,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58472","host":"gitlab.mydomain.com","headers":{"Accept":["application/json"],"Content-Type":["application/json"],"Cdn-Loop":["cloudflare"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f35f0700000e049d9b5f200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef293e9c4e049-DFW"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002866459,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:22 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271863.8905811,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58460","host":"gitlab.mydomain.com","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Content-Type":["application/json"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Cf-Ray":["5b5ef29d4f790f22-DFW"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Request-Id":["040f35f64b00000f221426e200000001"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["518"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002798531,"headers":{"Date":["Mon, 20 Jul 2020 19:04:23 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"]},"status":204}
{"level":"debug","ts":1595271865.4025197,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58432","host":"gitlab.mydomain.com","headers":{"Accept-Encoding":["gzip"],"Accept":["application/json"],"Content-Type":["application/json"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ray":["5b5ef2a6a96be049-DFW"],"Cf-Request-Id":["040f35fc2c0000e049d9826200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002709843,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:25 GMT"]},"status":204}
{"level":"debug","ts":1595271866.8915274,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58470","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Cf-Request-Id":["040f36020300000f22143b1200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"Cf-Ray":["5b5ef2b00b580f22-DFW"],"Content-Length":["518"],"Accept":["application/json"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Content-Type":["application/json"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002840086,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:26 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271868.4047647,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58442","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Cf-Ray":["5b5ef2b9690ee049-DFW"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["518"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Cf-Request-Id":["040f3607e20000e049d9905200000001"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Content-Type":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002585908,"headers":{"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:28 GMT"],"Gitlab-Ci-Builds-Polling":["yes"]},"status":204}
{"level":"debug","ts":1595271869.894527,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58444","host":"gitlab.mydomain.com","headers":{"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Connecting-Ip":["136.50.25.2"],"Cdn-Loop":["cloudflare"],"Content-Length":["518"],"Accept":["application/json"],"X-Forwarded-Proto":["https"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef2c2c9130f22-DFW"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f360dbd00000f221409d200000001"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002791285,"headers":{"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:29 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"]},"status":204}
{"level":"debug","ts":1595271871.3935568,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58468","host":"gitlab.mydomain.com","headers":{"Accept-Encoding":["gzip"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Cf-Ipcountry":["US"],"Content-Type":["application/json"],"Cf-Request-Id":["040f3613980000e049d99c2200000001"],"Cdn-Loop":["cloudflare"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Cf-Ray":["5b5ef2cc292ce049-DFW"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002688759,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:31 GMT"]},"status":204}
{"level":"debug","ts":1595271872.8924327,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58444","host":"gitlab.mydomain.com","headers":{"Accept":["application/json"],"Accept-Encoding":["gzip"],"Cf-Ray":["5b5ef2d58d5e0f22-DFW"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f36197500000f221415d200000001"],"Cf-Connecting-Ip":["136.50.25.2"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Content-Type":["application/json"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["518"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002826285,"headers":{"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:32 GMT"]},"status":204}
{"level":"debug","ts":1595271874.3962016,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58498","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f361f540000e049d9a91200000001"],"Cf-Ray":["5b5ef2deea85e049-DFW"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Accept":["application/json"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Type":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002889189,"headers":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:34 GMT"],"Gitlab-Ci-Builds-Polling":["yes"],"Strict-Transport-Security":["max-age=31536000"]},"status":204}
{"level":"debug","ts":1595271875.893535,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.1682.252:3021","request":{"method":"POST","uri":"/api/v4/jobs/request","proto":"HTTP/1.1","remote_addr":"127.0.0.1:58574","host":"gitlab.mydomain.com","headers":{"Cf-Ipcountry":["US"],"User-Agent":["gitlab-runner 13.1.0 (13-1-stable; go1.13.8; linux/amd64)"],"Cf-Ray":["5b5ef2e84de30f22-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["040f36252d00000f221428e200000001"],"Content-Type":["application/json"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["136.50.25.2, 127.0.0.1"],"Content-Length":["518"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Cf-Connecting-Ip":["136.50.25.2"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"","proto_mutual":true,"server_name":"gitlab.mydomain.com"}},"duration":0.002878833,"headers":{"Strict-Transport-Security":["max-age=31536000"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Server":["nginx"],"Date":["Mon, 20 Jul 2020 19:04:35 GMT"],"Gitlab-Ci-Builds-Polling":["yes"]},"status":204}
francislavoie commented 4 years ago

This seems to me like some Cloudflare misconfiguration more than anything, frankly. Unfortunately I don't have experience with Cloudflare so I don't have any suggestions.

Don-Swanson commented 4 years ago

except, I cannot get caddy to serve up anything either if I go directly to the IP/Port that it's running on (thus completely bypassing cloudflare)

Don-Swanson commented 4 years ago

I was able to get it up and running in a VM this weekend, and so far it seems stable. So what would be our next step?

(Side note, the VM version appears like it may be leaking my private IP address sometimes)

francislavoie commented 4 years ago

(Side note, the VM version appears like it may be leaking my private IP address sometimes)

What makes you say that? Please be specific.

So if it seems stable, it sounds like the issue was with your environment and that's now resolved. Since we never received any clear evidence as to the cause of the issue, there's nothing actionable in this issue so I'll close it.

Don-Swanson commented 4 years ago

The issue is not resolved. I still have issues when running it as a container. I simply did a test in a VM like you requested. I still want to run this containerized.

mholt commented 4 years ago

Well, if it works without a container, then I think it's pretty clear that it's an issue with the container's setup, not with Caddy. The forums will be better equipped to help you debug the container infrastructure (I myself am not a Docker user, so I won't be of much help here).

Don-Swanson commented 4 years ago

I get that, but I am using a container provided by Caddy.... ergo this is plausibly still a Caddy issue.

mholt commented 4 years ago

If it is a problem with the container we're providing, we're down for fixing it -- how can we reproduce it? As far as we can tell, you're the only one experiencing the issue, with millions of other deployments not having any problems. I know this sounds difficult -- and it's not easy -- but the burden is on you to help us reproduce it. Right now we can just guess (and I can't even do that since I don't have any Docker experience) and that won't get us anywhere... so we're depending on you to help us be able to experience the same behavior you're seeing. Until then, we have to assume it's something wrong with your particular environment/setup.

This requires a lot of effort, yes -- we also put a lot of effort into building and maintaining the software, and fixing the bugs we can reproduce. It's a two-way effort, so we need your help here.

francislavoie commented 4 years ago

Like I said, we don't understand the problem and we can't replicate it, and because we weren't given clear evidence, there's nothing we can do.

I'm not sure what you expect us to do at this point, we can't magically fix problems we don't understand.

Don-Swanson commented 4 years ago

I get the fact that y'all can't fix what you cannot reproduce, But like I've said before, what evidence can I provide? I've given you what logs I have. If you want more from me, you have to tell me what it is that you want.

mholt commented 4 years ago

I get the fact that y'all can't fix what you cannot reproduce, But like I've said before, what evidence can I provide? I've given you what logs I have. If you want more from me, you have to tell me what it is that you want.

We just want instructions. We gave you a template a long time ago up above. This is where you have to do some detective work, and really work hard to apply your pro troubleshooting skills. The usual process: break the problem down into parts, start as simple as possible, then add pieces one at a time until you recreate your setup, test it between each change, see where it starts breaking, that kind of thing.

Don-Swanson commented 4 years ago

I've done this. I've narrowed it down to when it makes connections to just my octoprint server that caddy crashes. But I also tested it against a known good octoprint server and it happens too (in the Caddy container). This only appears to have happened after a recent Caddy Container update. I've given you my caddy setup. I get that y'all are busy and that OpenSouce is all about contributions and that nobody is required to do anything. I'm not demanding support, because I hate when people do that to me, but I just keep getting the feeling from you that you don't have any desire to help me because I'm a one in a million use case and nobody else is having the same issue.

If you have no desire to help me, just tell me, I'll accept that and move on. I won't get angry or hold grudges. I just don't like being strung along without being given definitive requests. If you do want to try to help me, tell me exactly what you need from me. If I missed something in the template, tell me, we're all human and make mistakes. I could very well be 100% at fault, and if I am, I'd like to learn from it. I appreciate what y'all are doing for the community, but please let me know if you need additional things from me what exactly I'm missing or just tell me to go away, I have a hard time reading between the lines.

francislavoie commented 4 years ago

Could you try again with the debug global option added? It should reveal some additional logs that might help.

francislavoie commented 4 years ago

Also I will say those assumptions about our intentions are very rude. We're absolutely looking to improve this project, and any bug fix fits within that umbrella, but we're not magicians.

mholt commented 4 years ago

@TheEagle13

If you have no desire to help me, just tell me, I'll accept that and move on.

We want to help you and we want to fix any problems. But right now all we have is "it crashes" but we have no idea how to cause the crash. While it's good that you have shared your configs and stuff, we need your help to identify what kinds of things trigger the crash. You don't even have to look at the source code or anything, but we can't do this all ourselves. The point of open source is not "This is happening, fix it", it's "This is happening, I've figured out that this sequence of requests [or whatever] triggers it, what do you think?" and then we say "Ah yeah, that is a problem, we'll look into it and patch it up." But right now, we have no idea. We simply need you to find out what triggers it so we can reproduce it. If we can't reproduce it, we can't fix it. Like trying to throw darts blindfolded but hit the bullseye.

So, please try to meet us halfway, okay? Since there is no money being exchanged here, the currency of open source is time, hard work, and effort.

I just keep getting the feeling from you that you don't have any desire to help me because I'm a one in a million use case and nobody else is having the same issue.

If we didn't have any desire to help you, we wouldn't be pressing for more information. I promise. This is not personal! Leave the attitude behind and let's work as professionals here.

This only appears to have happened after a recent Caddy Container update. I've given you my caddy setup.

Which container update causes it? Try using older versions and see where the problem starts appearing. If it's consistent, maybe this should be moved to the container repo.

mholt commented 4 years ago

My apologies if any of this came off too demanding or harsh. I'm grateful you want to make it work, and we do too! It's OK if you don't feel like working on it right now; feel free to come back to it later if that will help.