[change] Stop sending checksum requests on 404

[nemesifier]

If a device gets deleted from the server but stays turned on will keep sending HTTP requests unnecessarily to the server, which is undesirable.

We should deal with this.

I think that if a 404 is detected, we could either stop the agent (exit with a non zero exit code) right away or let it try a few times before giving up.

[nemesifier]

@okraits @feckert any thoughts on this?

[Kevinjil]

This could create issues in orchestrated environments where services can return a 404 in some situations, such as https://github.com/traefik/traefik/issues/1689.

[nemesifier]

This could create issues in orchestrated environments where services can return a 404 in some situations, such as traefik/traefik#1689.

@Kevinjil thanks for your input.

That sounds like a bug in Traefik, since it would be better to return something different than 404 in those cases.

However we have to keep those temporary issues in mind, here's what I think we can do:

• if a checksum request returns 404, retry X times with a delay of 6 seconds, X being configurable and defaulting to 10, which results in 10 attempts over the span of a minute, hopefully any temporary reconfiguration of services in orchestrated environment lasts less than that, alternatively the user can configure a higher X
• if all retries fail, the agent will send a critical error log message and exit