samizdapp / herakles

slaying the hydra of IP/NAT/DNS/TLS/AppStore
2 stars 1 forks source link

Caddy often unresponsive #7

Open JoshuaCWebDeveloper opened 2 years ago

JoshuaCWebDeveloper commented 2 years ago

Problem:

Quite often, attempts to load Pleroma will timeout due to requests from the browser receiving no responses from Caddy.

Sometimes, this means that requests to http://setup.local timeout. When this happens, no errors (or logs from anywhere) are generally observed.

Other times, requests to http://setup.local will respond successfully, but requests to the Pleroma API timeout.

Most recently, requests to the Pleroma API timing out were accompanied by these errors:

 yggdrasil  Traceback (most recent call last):
 yggdrasil    File "/crawler/example.py", line 12, in <module>
 yggdrasil      ygg = YggdrasilConnection.fromServer()
 yggdrasil    File "/crawler/yggdrasil_iface.py", line 56, in fromServer
 yggdrasil      return cls(s.AF_INET, (host, port))
 yggdrasil    File "/crawler/yggdrasil_iface.py", line 40, in __init__
 yggdrasil      self.socket.connect(address)
 yggdrasil  ConnectionRefusedError: [Errno 111] Connection refused
 daemon_proxy  caught error TypeError: Cannot read properties of null (reading 'arrayBuffer')
 daemon_proxy      at PocketProxy.handleEvent (/proxy/dist/pocket_proxy.js:168:29)
 daemon_proxy      at runMicrotasks (<anonymous>)
 daemon_proxy      at processTicksAndRejections (node:internal/process/task_queues:96:5)

Usually, this issue will go away after awhile and requests will start responding again. A restart of the box/services doesn't necessarily help.

The overall result of all of this is that the box is not highly available and access to Pleroma is not reliable.

Expectation

Requests made to the box should result in a response. Both Caddy and the Proxy should be able to handle the failure of services they forward requests to.

JoshuaCWebDeveloper commented 2 years ago

I've observed that when the Pleroma API requests hang, Caddy is not receiving the requests or even the Websocket connection.

This is likely because the service worker is attempting to connect to setup.local:4000, but Caddy is listening on port 80.

JoshuaCWebDeveloper commented 2 years ago

This is likely because the service worker is attempting to connect to setup.local:4000, but Caddy is listening on port 80.

Caddy is listening on port 4000, so that's not correct.

However, I discovered that daemon_proxy was not starting the server up due to a bug:

[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy] Error: ENOENT: no such file or directory, open './upnp/addresses'
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.openSync (node:fs:594:3)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.readFileSync (node:fs:462:35)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at main (/proxy/proxy_server.js:6:8)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.<anonymous> (/proxy/proxy_server.js:14:1)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Module._compile (node:internal/modules/cjs/loader:1126:14)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Module.load (node:internal/modules/cjs/loader:1004:32)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Function.Module._load (node:internal/modules/cjs/loader:839:12)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]     at node:internal/main/run_main_module:17:47 {
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   errno: -2,
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   syscall: 'open',
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   code: 'ENOENT',
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy]   path: './upnp/addresses'
[Logs]    [8/27/2022, 9:35:01 AM] [daemon_proxy] }

Additionally, it looks like the hanging issue is actually with the proxy on the service worker side. For some reason, when the websocket connection fails, all of it's request hang instead of erroring. I couldn't figure out exactly what it is going on here as I had some problems running the next app locally.

JoshuaCWebDeveloper commented 2 years ago

The issue of running next locally is being tracked here: https://github.com/samizdapp/herakles/issues/8

JoshuaCWebDeveloper commented 2 years ago

Not sure if this is related or not, but I'll start with reporting it here.

Currently, Caddy is responding to the initial setup.local/ request, but it is responding with an empty body:

[Logs]    [9/1/2022, 1:30:09 PM] [daemon_caddy] 2022/09/01 20:30:09.822 INFO    http.log.access handled request {"request": {"remote_ip": "192.168.1.3", "remote_port": "1101", "proto": "HTTP/1.1", "method": "GET", "host": "setup.local", "uri": "/", "headers": {"Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Dest": ["document"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"], "Sec-Fetch-Mode": ["navigate"], "Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Ch-Ua": ["\"Chromium\";v=\"104\", \" Not A;Brand\";v=\"99\", \"Google Chrome\";v=\"104\""], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-User": ["?1"]}}, "user_id": "", "duration": 0.000163046, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}
JoshuaCWebDeveloper commented 2 years ago

Currently, Caddy is responding to the initial setup.local/ request, but it is responding with an empty body:

When Caddy responds with an empty body, it is often because no handlers matched the request.