Add Health checks

[vordimous]

Describe the desired outcome from the user's perspective As a DevOps maintainer optimizing Zilla deployments, I would expect Zilla to expose the status of its internal connections and services. If Zilla can determine if a downstream service is unreachable or degraded, It could provide an API to report the status. Any internal systems that can be in a Green, Yellow, or Red state would be helpful in reporting that status to an orchestration layer.

Acceptance criteria

• zilla can enable features through config or flag to report status
• endpoints meet existing standard formats

Additional context

• The closet thing to an HTTP standard spec I could find Health Check Response Format for HTTP APIs

• K8s liveliness/readiness/startup probes

• good blog post A guide to API health check

• Example cli health check from Redpanda rpk

• [ ] https://github.com/aklivity/zilla/issues/1131

[vordimous]

As a workaround until this feature is completed here is an example docker and bash health check for an active port. It is a convenient way to verify local service port listening status for docker health check just using bash.

In docker compose:

    healthcheck:
      interval: 5s
      timeout: 3s
      retries: 5
      test: ["CMD", "bash", "-c", "echo -n '' > /dev/tcp/127.0.0.1/1883"]

in terminal:

% bash -c "echo -n '' > /dev/tcp/localhost/8021"
bash: connect: Connection refused
% bash -c "echo -n '' > /dev/tcp/localhost/8022"
bash: /dev/tcp/localhost/8022: Connection refused