search

ItsEcholot / ContainerNursery

Puts Docker Containers to sleep and wakes them back up when they're needed
MIT License
94 stars 6 forks source link

CN is stuck in an infinite loop (almost denial-of-service like situation) for certain applications #56

Open accforgithubtest opened 7 months ago

accforgithubtest commented 7 months ago

Does CN need additional configurations for -

  1. Setting a "frequency" for "Checked if target is ready" (check every x seconds or milliseconds)
  2. Stop checking after X attempts and let the user manually reload the page (avoid the infinite requests situation).

Because, looks like CN is basically Getting into a infinite requests loop situation for certain applications, and this is happening indefinitely until CN container is restarted manually. And because I have a pi-hole in my setup, the pihole container is seeing crazy amounts of traffic from CN.

I have also tried using the proxyUseCustomMethod: GET, and it doesn't seem to help.

When manually looking at the docker logs for these applications, they are all starting up fine. Some start up really fast (Dozzle, FileBrowser) and some of them are slightly slower (code-server, homeAssistant), but even these are only a matter of about 3-8 seconds on a desktop.

In all these cases, I do not understand why CN is not able to recognize the container is up, because as soon as I restart CN the request for all of them is going thru successfully.
The issue seems to be when CN is starting the container. If the container is already running (i.e when CN is restarted), there is no issue it seems.

Applications I have seen this issue with - FileBrowser, dozzle, codeserver (vscode), n8n, HomeAssistant, etc (probably many more that I havent come across).

Debug Logs for File Browser - 

2024-01-31T06:49:08.056936749Z DEBUG [2024-01-31 15:49:07.932 +0900]: Checked if target is ready
2024-01-31T06:49:08.056978408Z     domain: [
2024-01-31T06:49:08.056993543Z       "files.local.host"
2024-01-31T06:49:08.057006083Z     ]
2024-01-31T06:49:08.057017366Z     proxyHost: "filebrowser"
2024-01-31T06:49:08.057029250Z     proxyPort: 80
2024-01-31T06:49:08.057041190Z     status: 404
2024-01-31T06:49:08.057053335Z     headers: {}
2024-01-31T06:49:08.184837823Z DEBUG [2024-01-31 15:49:08.178 +0900]: Resetting connection timeout
2024-01-31T06:49:08.184854245Z     domain: [
2024-01-31T06:49:08.184859161Z       "files.local.host"
2024-01-31T06:49:08.184863442Z     ]
2024-01-31T06:49:08.184867404Z     timeoutSeconds: 300
2024-01-31T06:49:08.185054805Z DEBUG [2024-01-31 15:49:08.180 +0900]: Checked if target is ready
2024-01-31T06:49:08.185067614Z     domain: [
2024-01-31T06:49:08.185072158Z       "files.local.host"
2024-01-31T06:49:08.185075821Z     ]
2024-01-31T06:49:08.185079285Z     proxyHost: "filebrowser"
2024-01-31T06:49:08.185082752Z     proxyPort: 80
2024-01-31T06:49:08.185086191Z     status: 404
2024-01-31T06:49:08.185089660Z     headers: {}
2024-01-31T06:49:08.441863373Z DEBUG [2024-01-31 15:49:08.429 +0900]: Resetting connection timeout
2024-01-31T06:49:08.441911183Z     domain: [
2024-01-31T06:49:08.441926747Z       "files.local.host"
2024-01-31T06:49:08.441939997Z     ]
2024-01-31T06:49:08.441952086Z     timeoutSeconds: 300
2024-01-31T06:49:08.442559338Z DEBUG [2024-01-31 15:49:08.433 +0900]: Checked if target is ready
2024-01-31T06:49:08.442616632Z     domain: [
2024-01-31T06:49:08.442643761Z       "files.local.host"
2024-01-31T06:49:08.442661863Z     ]
2024-01-31T06:49:08.442673794Z     proxyHost: "filebrowser"
2024-01-31T06:49:08.442685461Z     proxyPort: 80
2024-01-31T06:49:08.442697545Z     status: 404
2024-01-31T06:49:08.442709196Z     headers: {}
2024-01-31T06:49:08.699979206Z DEBUG [2024-01-31 15:49:08.679 +0900]: Resetting connection timeout
2024-01-31T06:49:08.700044822Z     domain: [
2024-01-31T06:49:08.700094337Z       "files.local.host"
2024-01-31T06:49:08.700108119Z     ]
2024-01-31T06:49:08.700119896Z     timeoutSeconds: 300
2024-01-31T06:49:08.700818779Z DEBUG [2024-01-31 15:49:08.685 +0900]: Checked if target is ready
2024-01-31T06:49:08.700859590Z     domain: [
2024-01-31T06:49:08.700875042Z       "files.local.host"
2024-01-31T06:49:08.700887587Z     ]
2024-01-31T06:49:08.700898977Z     proxyHost: "filebrowser"
2024-01-31T06:49:08.700910679Z     proxyPort: 80
2024-01-31T06:49:08.700926719Z     status: 404
2024-01-31T06:49:08.700946391Z     headers: {}

Debug Logs for dozzle - 

DEBUG [2024-01-31 16:04:51.746 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}
DEBUG [2024-01-31 16:04:51.995 +0900]: Resetting connection timeout
    domain: [
      "logs.local.host"
    ]
    timeoutSeconds: 600
DEBUG [2024-01-31 16:04:51.999 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}
DEBUG [2024-01-31 16:04:52.246 +0900]: Resetting connection timeout
    domain: [
      "logs.local.host"
    ]
    timeoutSeconds: 600
DEBUG [2024-01-31 16:04:52.249 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}
DEBUG [2024-01-31 16:04:52.497 +0900]: Resetting connection timeout
    domain: [
      "logs.local.host"
    ]
    timeoutSeconds: 600
DEBUG [2024-01-31 16:04:52.501 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}
DEBUG [2024-01-31 16:04:52.707 +0900]: Proxied request
    host: "logs.local.host"
    target: {
      "protocol": "http:",
      "host": "127.0.0.1",
      "port": 8080
    }
    headers: {
      "x-container-nursery-container-name": "dozzle"
    }
DEBUG [2024-01-31 16:04:52.747 +0900]: Resetting connection timeout
    domain: [
      "logs.local.host"
    ]
    timeoutSeconds: 600
DEBUG [2024-01-31 16:04:52.751 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}
DEBUG [2024-01-31 16:04:52.998 +0900]: Resetting connection timeout
    domain: [
      "logs.local.host"
    ]
    timeoutSeconds: 600
DEBUG [2024-01-31 16:04:53.002 +0900]: Checked if target is ready
    domain: [
      "logs.local.host"
    ]
    proxyHost: "dozzle"
    proxyPort: 8080
    status: 405
    headers: {}

I also have few other containers where I do not see this issue. For example, MkDocs is another container I have which does take over 5 seconds to start up, however CN handles it well without any issues. Its only certain containers above that have this issue.

accforgithubtest commented 7 months ago

Another example - for sqlpage, which runs a few sql's and takes about 20-25 seconds to load because of multiple SQL. In this case, CN is hitting the application with so many requests, that MySQL connection pool is getting exhausted.

image

This is where some controls around the application behaviour might help.

  1. Setting a "frequency" for "Checked if target is ready" (check every x seconds or milliseconds)
  2. Stop checking after X attempts and let the user manually reload the page (avoid the infinite requests situation).
ItsEcholot commented 7 months ago

Regarding your first post: This looks like a misconfigured network / proxyHost value to me. Please ensure you can ping / curl the domain filebrowser or dozzle from inside the container nursery container.

Regarding your second post: This seems a bit like an application issue to me... ContainerNursery tries to generate as little load as possible for the target service by using HEAD requests (per default). If an application can be crashed by simply requesting it a few times per second, it might just be a badly configured / coded application (for example, missing / wrong limit on connections / threads which is set to same as db connection pool size). That being said I agree with your point that there might be value in adding an option to configure how often ContainerNursery retries the page.

Do you have time to create a pull request for this? If not, I can take a look at this some time but I can't give you a timeframe.

accforgithubtest commented 7 months ago

Hi !, thanks for your reply.

The ping logs from within the CN container shows that it is able to reach the filebrowser container without any issues.

Apologies If I dont understand this correctly, can you please help me understand what / how network configuration can be an issue here, because as soon as I restart the container, I am able to reach the applications via the url. Also the CN config file directly uses the container name, there is no other network specific configuration (apart from pi-hole that points the local dns - which I am sure is working fine as there are other applications that CN works with perfectly fine like handbrake, MkDocs, etc).

- domain:
      - files.local.host
    containerName:
      - filebrowser
    proxyHost: filebrowser
    proxyPort: 80
    timeoutSeconds: 1800
    proxyUseCustomMethod: GET

The ping logs from within the CN container shows that it is able to reach the filebrowser container. 

/usr/src/app # ping files.local.host
PING files.local.host (192.168.100.25): 56 data bytes
64 bytes from 192.168.100.25: seq=0 ttl=64 time=0.050 ms
64 bytes from 192.168.100.25: seq=1 ttl=64 time=0.146 ms
64 bytes from 192.168.100.25: seq=2 ttl=64 time=0.144 ms
64 bytes from 192.168.100.25: seq=3 ttl=64 time=0.142 ms
64 bytes from 192.168.100.25: seq=4 ttl=64 time=0.146 ms
64 bytes from 192.168.100.25: seq=5 ttl=64 time=0.146 ms
64 bytes from 192.168.100.25: seq=6 ttl=64 time=0.145 ms
64 bytes from 192.168.100.25: seq=7 ttl=64 time=0.064 ms
^C
--- files.local.host ping statistics ---
8 packets transmitted, 8 packets received, 0% packet loss
round-trip min/avg/max = 0.050/0.122/0.146 ms
/usr/src/app #
ItsEcholot commented 7 months ago

Since you ping'ed files.local.host and not filebrowser you should probably set proxyHost to files.local.host, or alternatively check if pinging using filebrowser also works (which it should, if both containers are in the same docker network).

If pinging works please also try curling http://filebrowser:80 from the ContainerNursery container.

accforgithubtest commented 7 months ago

Thanks for the reply @ItsEcholot .

So, while the web browser is still stuck on the "waking up files" page, I ran the curl command from within CN container and getting the response from filebrowser (looks to be the html for the page file browser UI).

/usr/src/app # curl http://filebrowser:80
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, user-scalable=no"
    />

    <title>
      File Browser
    </title>

    <link
      rel="icon"
      type="image/png"
      sizes="32x32"
      href="/static/img/icons/favicon-32x32.png"
    />

.......... (lots more html)

So it looks like even though the filebrowser container is up and ping / curl commands are responding correctly, the UI is still stuck in the "Waking up" page.