s6 service start up improvements

[matthewvalimaki]

Images based on alpine-consul-base do not seem to specify s6 start order potentially leading to unwanted behavior. If I have understood s6 correctly there is no actual support for setting an order but there is a concept of dependencies, see Readiness notification and dependency management and more specifically The s6-svwait program.

Example use case:

Scenario: I want all services to be discoverable through Consul server.
Given I use `alpine-consul` for service discovery
And `alpine-consul-nginx` for web server
When I start `alpine-consul-nginx`
Then Service `nginx` should be discoverable and `alive` via Consul

However, as it is today there's no guarantee that nginx indeed is up before Consul agent reports "nginx" service. While default Consul configuration for nginx does have health check against /ping, nginx might not even start up. Therefore: 1) Reporting service early I think is bad practice (why report if it never can be used) 2) /ping might not be available.

Regarding 1. I do acknowledge that not reporting service can lead to a situation where health monitoring breaks if it depends on information provided by Consul. Perhaps instead of exit in my example below we could do a log entry instead and proceed with exec consul as before.

Change proposals: 1) Have Consul s6 run script wait for nginx (node etc.) to become available. If it does not become available, never launch Consul.

Note: example below does not contain -t for timeout, probably should be defined to 10s for nginx.

# wait for nginx to be up
s6-svwait -u /var/run/s6/services/nginx/

if [ $? -ne 0 ]; then
  exit $?;
fi

exec consul ...

2) Replace nginx /ping health check with The s6-svstat program s6-svstat /var/run/s6/services/nginx, see exit codes.

[smebberson]

I usually bring up my services in a down state, and then once they're ready switch everything over as necessary. I designed it so that Consul would be connected as soon as possible. If it reported early that a service was down then nothing would necessarily change, and therefore no downside.

I'll check into s6-svstat a little more. The exit codes don't seem to provide enough information to determine if it's up or down. For example, can an exit code of 0 mean down and ready AND up and ready? If so, inspecting the result would be necessary and I'm not sure that's a great idea. I guess we'll have to ask on the s6 mailing list for more info.

All of these images would designed so that you could use your Dockerfile to replace any of the inner working as you see fit, and as few at a time as possible. There are obviously many different requirements for many different stacks out there. So if you wanted to not bring Consul up until nginx was ready, you could simply alter the run script for Consul as you have it. Same thing goes for the /ping check. The /ping could be customised to whatever best suits your environment.

I'll definitely look into s6-svstat a little more however. From memory, when I looked while designing this stuff, there didn't seem to be a lot of best practice around service discovery and nginx up/down monitoring on the web. Have you come across much?

[matthewvalimaki]

Good points. I might have wasted your time with my requests.

Provided use case was based on assumption not real life. Based on your explanation I do now feel that proposed change 1. brings no benefits.

My experience with s6 is limited. Reason why I made my recommendation was because I replaced default.conf and did not notice to bring /ping over. s6-svstat /var/run/s6/services/nginx/ gives status of master process therefore your points regarding the exit code are valid. Should have checked that before asking for a change.

I was going after a way where I wouldn't have to change the default Consul nginx.json but probably I will do that as the name is vague and then if I do that I could as easily change /ping like you said.

I'm closing this as there's nothing to do.

[smebberson]

@matthewvalimaki, no problems at all.

I kind of see the default nginx.json as throw away. It's there to explain how it all hangs together, but you'd likely want to do something different in your real world scenario.

When I use it in real life, I don't have a /ping route either, but something slightly different.