kafka wait for topics?

[nseyvet]

This is not an issue, rather a question.

In many containers dependent on kafka, there is a check in the entrypoint.sh (or start.sh or equivalent docker entrypoint file) to wait for Kafka to be up and running and the topics created.

E.g. monasca-thresh/submit.sh

if [ -n "$KAFKA_WAIT_FOR_TOPICS" ]; then
  echo "Waiting for Kafka topics to become available..."
  success="false"

  for i in $(seq $KAFKA_WAIT_RETRIES); do
    python $STORM_PY_HELPER_DIR/kafka_wait_for_topics.py
    if [ $? -eq 0 ]; then
      success="true"
      break
    else
      echo "Kafka not yet ready (attempt $i of $KAFKA_WAIT_RETRIES)"
      sleep "$KAFKA_WAIT_DELAY"
    fi
  done

  if [ "$success" != "true" ]; then
    echo "Kafka failed to become ready, exiting..."
    sleep 1
    exit 1
  fi
fi

My question: When is the fact that Kafka not available or topics not created going to prevent the overall system to work?

IMO, any components using Kafka client libraries do not require check like these.

For example, monasca-thresh (which is based on storm), surely, storm reconnects in case the subscribed topics disappear or get created. Or??

[kornicameister]

monasca does not rely on it and expects all the topic preconfigured prior to starting any components. That check above ensure that if the components finally starts it will be able to find the topics it requires to work properly, either to push data or to read it. Another point is that for most of the cases in monasca, if not every one of them, the lack of the topic inside the kafka is critical error hence the component shouldn't even start.

That's the backround, not really sure if storm works as you described and if the explanation is correct :D, that's just how I understand it.

@timothyb89 do you have anything to add ? @nseyvet does it help at all ?

[nseyvet]

The explanation helps but the following statement does not: « the lack of the topic inside the kafka is critical error hence the component shouldn't even start”.

It could be more resilient to allow the system to recover from a failed state rather than not to start.

Overall, the presence of a “wait for kafka” raises questions as to the system overall behavior once under load and components fail or move?

I would recommend to remove all(except Kafka-init) “wait for Kafka and topics” from the entry point files of the containers. Instead rely on either a) the container crashing and restarting or b) a resilient behavior from the Kafka library.

Additionally, maybe Kafka-init behavior could be moved to the Kafka container to ensure quick topic creation?

What do u think?

On Wed, 11 Oct 2017 at 11:29, Tomasz Trębski notifications@github.com wrote:

monasca does not rely on it and expects all the topic preconfigured prior to starting any components. That check above ensure that if the components finally starts it will be able to find the topics it requires to work properly, either to push data or to read it. Another point is that for most of the cases in monasca, if not every one of them, the lack of the topic inside the kafka is critical error hence the component shouldn't even start.

That's the backround, not really sure if storm works as you described and if the explanation is correct :D, that's just how I understand it.

@timothyb89 https://github.com/timothyb89 do you have anything to add ? @nseyvet https://github.com/nseyvet does it help at all ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/monasca/monasca-docker/issues/248#issuecomment-335751923, or mute the thread https://github.com/notifications/unsubscribe-auth/AdweXByT_5vYxLmYHKdWK6CnJEb8-AXDks5srIp7gaJpZM4Pz_XW .

[kornicameister]

Well, I can only tell sth about moving the logic into kafka image. That's far as long as you want to use kafka provided by monasca-docker. If you have your own kafka cluster, you have a small problem. Just small, because there would be workarounds around that. However the idea with -init containers is just a bit more flexible.

About the allowing to recover if the something out there is not ready, well I think this is an idea worth discussing. It would simplify the images a bit. The question I have is should we:

• aim at just updating compose files to allow restarting upon failures
• update the upstream components to allow them to wait patiently for everything to be ready

?

[nseyvet]

The -init is more flexible.

I would go for:

• aim at just updating compose files to allow restarting upon failures

On Wed, 11 Oct 2017 at 12:28, Tomasz Trębski notifications@github.com wrote:

Well, I can only tell sth about moving the logic into kafka image. That's far as long as you want to use kafka provided by monasca-docker. If you have your own kafka cluster, you have a small problem. Just small, because there would be workarounds around that. However the idea with -init containers is just a bit more flexible.

About the allowing to recover if the something out there is not ready, well I think this is an idea worth discussing. It would simplify the images a bit. The question I have is should we:

• aim at just updating compose files to allow restarting upon failures
• update the upstream components to allow them to wait patiently for everything to be ready

?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/monasca/monasca-docker/issues/248#issuecomment-335766893, or mute the thread https://github.com/notifications/unsubscribe-auth/AdweXAhsDu5Y255VUpcFq0deVbrivZeOks5srJhTgaJpZM4Pz_XW .

[kornicameister]

Let's see what others think about that ;-)

[timothyb89]

My feeling about restarting quickly on failure is that you tend to generate a lot of noise for a situation that you generally expect to happen.

As an example, Kubernetes keeps a restart counter for containers. Ideally a restart count of zero should imply (to a degree) that the container is healthy, and a positive restart count (especially if it's increasing) should imply that there is a problem that needs fixing. If we fail quickly on startup because some component hasn't finished initializing, the restart count no longer provides any hint that something is wrong with the system, as now we always expect the restart count to be > 0.

As for where that readiness check should be... generally I'd prefer that to be a part of the application itself, e.g. not part of our bash startup script. But we figured it would be simpler to add it to our container startup scripts than patch it into their respective upstream components, plus lots of components are 3rd party and can't be reasonably patched.

Some summarized thoughts:

• relying on container restart policies for readiness checks between components is not desirable
• we should configure everything to restart on failure (this is the default behavior in Kubernetes but not docker-compose)
• patching upstream components to wait would be nice (but I don't think it's a high priority)

[craigbr]

I agree with Tim that we shouldn't just continue restarting until the topic is ready. The load from that could cause other failures. For example, restarts of monasca-thresh are pretty expensive since it was not designed to be used as a container.

And we should restart on failures.

Is anyone planning on using docker-compose for real production? Would be interesting to know.

[timothyb89]

A couple of points I missed..

Overall, the presence of a “wait for kafka” raises questions as to the system overall behavior once under load and components fail or move?

I think the lack of a topic generally should be considered a critical error. At startup the best behavior is to try creating topics automatically and wait a reasonable amount of time before quitting (as we do now), but a situation like a topic disappearing at runtime is - per my understanding, at least - not something we can realistically recover from. Kafka should be pretty resilient, but a complete failure of its HA more or less requires human intervention.

Additionally, maybe Kafka-init behavior could be moved to the Kafka container to ensure quick topic creation?

We actually just moved kafka topic creation into a separate container a few weeks ago in #138 and #153 after discussion in #134. The reasons being:

• we give much better feedback to users by keeping jobs in separate containers, and don't need to do any manual job management (nasty in docker)
• there isn't a practical disadvantage to separating the topic init functionality, create-topics.sh works over the network (via zookeeper) regardless of whether kafka is running alongside it or not

[nseyvet]

Kafka-init as a separate container is a valid separation of concern. Let's park that topic :).

From my perspective, a positive restart count (kubernetes) that stops growing indicates eventual consistency. The current "wait" behaviour indicates a hard consistency requirement. Since this is micro-services, eventually big data, and a distributed system, it seems more natural to rely on eventual consistency than hard consistency. Hence, the current behavior gives an impression of brittleness of the ensemble.

There is an additional problem with the current "wait" implementation: It relies on Python not on bash/sh. Most of the Monasca components are Java based (ELK, Storm, Kafka, etc.), then some are in Python, and some in Go. But, currently, all containers need to package Python in the image.

Another side effect is the two separate processes in the container one in Java/Python/Go (what I want to run) and one in Python(that does the wait). Which also feels weird.

Finally, when the system runs under load, it is a guarantee that Kafka will fail or that components will lose connectivity. At that point, the overall Monasca must recover without human intervention.

To sum up, while I understand the arguments, and also measure that the "wait for topics" is a small thing, the current liveness ("wait for topics") check implementation feels IMO like a bad implementation from a container, micro-service, resiliency perspective. Relying on eventual consistency feels better and simpler.

(Plus most of the components handle eventual consistency well so far.)

What are best practices for liveness check across distributed application in a container world?

[matrixik]

At some point this could be tested in Kubernetes with tool like https://github.com/asobti/kube-monkey

[timothyb89]

There is an additional problem with the current "wait" implementation: It relies on Python not on bash/sh. [...] But, currently, all containers need to package Python in the image.

Including a Python interpreter is admittedly somewhat less than ideal, but seems like a reasonable compromise since we're inevitably going to need to share several utility functions between all our containers, e.g.

• config file templating via jinja2 (this is enough of a benefit by itself in many cases, IMO)
• generic tcp wait
• kafka wait
• the mini kubernetes library
• the jvm heap scaler
• misc app-specific utility scripts (e.g. kafka_mirror.py, storm's clean_externals.py, etc)

(admittedly we do need to make improvements with how that utility code is shared between all our containers - it should mostly move to monasca/python now)

Another side effect is the two separate processes in the container one in Java/Python/Go (what I want to run) and one in Python(that does the wait). Which also feels weird.

IMO the important thing is to avoid any complex process management in containers (e.g. requiring supervisord or similar). Running multiple processes is nearly unavoidable if you have any startup logic whatsoever, even just setting up config files will probably require a sed call or two. That said, in almost no(*) cases do our containers actively run more than one process at once.

(* except keystone, but please don't use our keystone container in production)

Relying on eventual consistency feels better and simpler.

Generally speaking, I agree. If Monasca had been designed from the ground up to operate in a container and microservice world, our containers would be substantially simpler since all of this extra logic would just be built in to each component. Alas, we are retrofitting old components. :slightly_smiling_face:

What are best practices for liveness check across distributed application in a container world?

One common practice is to expose some health check endpoint over HTTP, most Kubernetes components expose /healthz for this purpose. For 'old'-style applications like Kafka, though, you need to do something application specific. Docker recommends a simple TCP connection check or some app-specific check.

[matrixik]

JFYI for monasca-log-agent (that use Logstash = Ruby) I avoided installing entire Python thanks to https://github.com/wrouesnel/p2cli tool. I just download one self contained binary https://github.com/monasca/monasca-docker/blob/master/monasca-log-agent/Dockerfile#L36 (6 MB binary).

[nseyvet]

p2cli is a nice reference!

To remove python dependencies, confd is being used in our Dockerfiles (~16MBs if I remember correctly).

[timothyb89]

p2cli does look nice! I've really come to appreciate jinja2-style templates over go's builtin templates after working with Helm, and that utility would be a nice way to slim down some of our images. I just filed #260 to look at switching grafana over to using it.