Feature request: Notification for updated services

[audunhalland]

Use case: I have docker-flow-proxy and docker-flow-swarm-listener running, managing proxying into my own docker services (defined in a separate swarm stack file). A CI job redeploys the stacks whenever we change stack files. Sometimes we change com.df.* labels for an already existing service, and expect DFP to get reconfigured automatically, but it doesn't. It only works for newly added services.

According to documentation on https://swarmlistener.dockerflow.com/:

At the moment, the only supported option is to send a notification when a new service is created, or an existing service was removed from the cluster.

So it looks like this is not a bug, rather it's just never been implemented. Also the DFSL env var DF_NOTIFY_CREATE_SERVICE_URL seems to indicate that only newly created services are supported.

The least inconvenient workaround for us right now is just to manually scale DFP to 0 and then back to 2 after the CI job has finished running. We could move this step into CI, but I think we can minimize downtime further if the running proxy just reconfigured itself instead of a hard service restart.

[thomasjpfan]

The website is incorrect. DFSL does send create notifications when there are changes in com.df.* labels.

Which version of DFSL are you using? Can you give an example of how you are experience this issue?

[thomasjpfan]

You can test out this feature by:

docker network create --driver overlay proxy

docker service create --name proxy \
    -p 80:80 \
    -p 443:443 \
    -p 8080:8080 \
    --network proxy \
    dockerflow/docker-flow-proxy

docker service create --name swarm-listener \
    --network proxy \
    --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
    -e DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure \
    -e DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove \
    --constraint 'node.role==manager' \
    dockerflow/docker-flow-swarm-listener

docker service logs swarm-listener -f

This will print the logs of DFSL. In another session, you can change the labels on the whoami service:

docker service update --label-add com.df.hello=world whoami

In the DFSL logs, you should be able to see the hello parameter change as you update the com.df.* labels:

docker service update --label-add com.df.hello=cows whoami

[audunhalland]

I tried this locally with docker for mac and it works well. However it is broken in my cloud environments (docker for AWS) and with your help I have roughly tracked it down to docker events not outputting any service events (in neither of our 3 swarm environments).

Greatly appreciated!

[thomasjpfan]

@audunhalland If there is a docker events issue, you can set the environmental variable: DF_SERVICE_POLLING_INTERVAL in DFSL to a positive number, such as 20. This tells DFSL to poll the docker api every 20 seconds for changes in your services.

[audunhalland]

Yes, I could try that, but polling is not that ideal, let's see. Thanks again.

[thomasjpfan]

@audunhalland There were past issues regarding docker events for docker version 18.03 (One of the reasons polling was added). Updating to 18.06 seems to resolve docker event issues.

[hatched-DavidMichon]

@thomasjpfan I am facing an issue that might be related to how events are managed when scaling service.

When I asked for more replicas for my service and there's no suitable node because of insufficient memory, no notification is generated by DFSL. This is problematic because I would like that my @replicas_running alerts in alertmanager to be updated to reflect the change of expected replicas.

I can see however a service update events in the docker events API like this:

2018-11-13T13:28:55.244973820Z service update bo3sfovtc0om4gvf73suf3zxg (name=staging-cookbook_cookbook, replicas.new=3, replicas.old=1)

In this case I would expect DSFL to send a notification.

Note That if I increase service replicas and there's enough suitable nodes, DFSL send the notification correctly.

[thomasjpfan]

@hatched-DavidMichon Currently, DFSL waits for the service to converge before sending out the notification. In your case, since there are not enough memory, the service never converges, thus DFSL does not send out this notification. DFSL waits for services to converge, allowing for DFP to be more stable and not receive services that have not come up yet.

Given that, I can see a few ways to work around your issue. Both involves updating DFSL.Can you open up a new issue for your problem?

[hatched-DavidMichon]

@thomasjpfan I opened a new issue regarding this problem. Thanks for your prompt reply! At you disposal to discuss and test workaround solutions.