Caddy often unresponsive

[JoshuaCWebDeveloper]

Problem:

Quite often, attempts to load Pleroma will timeout due to requests from the browser receiving no responses from Caddy.

Sometimes, this means that requests to http://setup.local timeout. When this happens, no errors (or logs from anywhere) are generally observed.

Other times, requests to http://setup.local will respond successfully, but requests to the Pleroma API timeout.

Most recently, requests to the Pleroma API timing out were accompanied by these errors:

 yggdrasil  Traceback (most recent call last):
 yggdrasil    File "/crawler/example.py", line 12, in <module>
 yggdrasil      ygg = YggdrasilConnection.fromServer()
 yggdrasil    File "/crawler/yggdrasil_iface.py", line 56, in fromServer
 yggdrasil      return cls(s.AF_INET, (host, port))
 yggdrasil    File "/crawler/yggdrasil_iface.py", line 40, in __init__
 yggdrasil      self.socket.connect(address)
 yggdrasil  ConnectionRefusedError: [Errno 111] Connection refused

 daemon_proxy  caught error TypeError: Cannot read properties of null (reading 'arrayBuffer')
 daemon_proxy      at PocketProxy.handleEvent (/proxy/dist/pocket_proxy.js:168:29)
 daemon_proxy      at runMicrotasks (<anonymous>)
 daemon_proxy      at processTicksAndRejections (node:internal/process/task_queues:96:5)

Usually, this issue will go away after awhile and requests will start responding again. A restart of the box/services doesn't necessarily help.

The overall result of all of this is that the box is not highly available and access to Pleroma is not reliable.

Expectation

Requests made to the box should result in a response. Both Caddy and the Proxy should be able to handle the failure of services they forward requests to.

[JoshuaCWebDeveloper]

I've observed that when the Pleroma API requests hang, Caddy is not receiving the requests or even the Websocket connection.

This is likely because the service worker is attempting to connect to setup.local:4000, but Caddy is listening on port 80.

[JoshuaCWebDeveloper]

This is likely because the service worker is attempting to connect to setup.local:4000, but Caddy is listening on port 80.

Caddy is listening on port 4000, so that's not correct.

However, I discovered that daemon_proxy was not starting the server up due to a bug:

[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy] Error: ENOENT: no such file or directory, open './upnp/addresses'
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.openSync (node:fs:594:3)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.readFileSync (node:fs:462:35)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at main (/proxy/proxy_server.js:6:8)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.<anonymous> (/proxy/proxy_server.js:14:1)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Module._compile (node:internal/modules/cjs/loader:1126:14)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Module.load (node:internal/modules/cjs/loader:1004:32)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Function.Module._load (node:internal/modules/cjs/loader:839:12)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at node:internal/main/run_main_module:17:47 {
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   errno: -2,
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   syscall: 'open',
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   code: 'ENOENT',
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   path: './upnp/addresses'
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy] }

Additionally, it looks like the hanging issue is actually with the proxy on the service worker side. For some reason, when the websocket connection fails, all of it's request hang instead of erroring. I couldn't figure out exactly what it is going on here as I had some problems running the next app locally.

[JoshuaCWebDeveloper]

The issue of running next locally is being tracked here: https://github.com/samizdapp/herakles/issues/8

[JoshuaCWebDeveloper]

Not sure if this is related or not, but I'll start with reporting it here.

Currently, Caddy is responding to the initial setup.local/ request, but it is responding with an empty body:

[Logs]    [9/1/2022, 1:30:09 PM] [daemon_caddy] 2022/09/01 20:30:09.822 INFO    http.log.access handled request {"request": {"remote_ip": "192.168.1.3", "remote_port": "1101", "proto": "HTTP/1.1", "method": "GET", "host": "setup.local", "uri": "/", "headers": {"Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Dest": ["document"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"], "Sec-Fetch-Mode": ["navigate"], "Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Ch-Ua": ["\"Chromium\";v=\"104\", \" Not A;Brand\";v=\"99\", \"Google Chrome\";v=\"104\""], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-User": ["?1"]}}, "user_id": "", "duration": 0.000163046, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}

[JoshuaCWebDeveloper]

Currently, Caddy is responding to the initial setup.local/ request, but it is responding with an empty body:

When Caddy responds with an empty body, it is often because no handlers matched the request.