sablierapp / sablier

Start your containers on demand, shut them down automatically when there's no activity. Docker, Docker Swarm Mode and Kubernetes compatible.
https://sablierapp.dev/
GNU Affero General Public License v3.0
1.46k stars 48 forks source link

dynamic/blocking does not work for some services #443

Open DasDunkel opened 2 weeks ago

DasDunkel commented 2 weeks ago

Describe the bug Honestly, I have no real idea what the cause of this could even be

One of the services I host (dumb) starts and stops as expected when a request is made, however, if dynamic is enabled the page is not shown, and if blocking is enabled the request is not blocked, instead the user has to refresh the page themselves once the service is ready.

You can test it here, session length is 5 seconds so you don't have to wait around; https://dumb.dunk.dev

Afaik it's not an issue with my overall configuration as it works for other services (example below) https://overflow.dunk.dev

Context

Expected behavior The service starting page is shown/connection hangs until service is ready

Additional context Setup in caddy;

route {
    sablier http://172.20.0.3:10000 {
            group dumb
            session_duration 5s
            dynamic {
                    display_name Dumb
            refresh_frequency 2s
            theme ghost
            }
    }
    reverse_proxy 172.20.0.107:5555
}
acouvreur commented 2 weeks ago

Hey @DasDunkel

What is the log output of Sablier ?

I think that maybe the session duration is so short that it is never ready.

Currently the session duration starts when calling start.

For the dynamic, you have to set the refresh rate lower than the session duration.

DasDunkel commented 2 weeks ago

Hi @acouvreur

Below is the output from sablier for each domain, I have tried increasing the session length to both 1 minute and 5 minutes, but the 'starting service' page still does not show

The service itself (dumb) starts almost immediately, which is the only real difference between other working services (besides the obvious stuff)

https://txt.dunk.dev/dunk/9e84c38c03f74a2b9e76922075cf8b58

Edit; Enabling debug logging just adds the following lines;


time="2024-11-07T23:14:07Z" level=debug msg="expiring {Name:Dumb CurrentReplicas:1 DesiredReplicas:1 Status:ready Message:} in 1m0s"
time="2024-11-07T23:14:07Z" level=debug msg="map[dumb:[Dumb] overflow:[AnonymousOverflow]]"
DasDunkel commented 2 weeks ago

After a bit of poking at random thoughts, and testing a few things, I think I figured it out, and it seems to be a weird edge case, let me try and explain;

The dumb container starts almost immediately, but it might take a second or two for it to be able to process incoming requests, so when sablier starts the container it seems to think it's instantly available, so instantly redirects, but because the service isn't fully ready, it returns an empty response.

I have tried adding a health check to the container which would ensure it's actually serving content, but unfortunately this service's container is so stripped down it doesn't even have bash/sh/ls, so I might need to figure out a way of doing that if something can't be done in sablier, maybe something like a forced initial wait option to give the service chance?

acouvreur commented 2 weeks ago

You could try contributing to the image and add a healthcheck command, but I understand the wait option in Sablier to avoid that.

Maybe an option like consider-ready-after: 10s what do you think?

DasDunkel commented 2 weeks ago

I'll definitely look into pushing a health check to dumb, but I think something like consider-ready-after would be a great addition, especially for some services where they can process requests while starting up, so would show as healthy, but are incredibly slow/unresponsive until they've finished things like startup tasks

And now that I think about it, I think dumb might be one of these types of services as I never got a 404/other error before it was ready to serve, but that's an issue for elsewhere.

Thanks for your time