search

RaimondB / Detecticam

Offers Neural Network Recognition (Yolov3) of IP Camerafeeds and signalling
GNU General Public License v3.0
7 stars 2 forks source link

Add observability to monitor detection rate, health and performance #1

Closed RaimondB closed 4 years ago

RaimondB commented 4 years ago

Add metrics and report them to (at least) influxdb.

Sensible metrics:

Using a framework such as https://www.app-metrics.io/reporting/reporters/influx-data/

jeroenhe commented 4 years ago

That would certainly be nice additions.

Not sure if it fits here, but what I'd like to have is the process offering a (HTTP API) health endpoint I can query using the HEALTHCHECK-functionality (part of the docker engine). This could help me detect and restart the container in case it doesn't function properly, according to its own provided API.

RaimondB commented 4 years ago

Not sure if it fits here, but what I'd like to have is the process offering a (HTTP API) health endpoint I can query using the HEALTHCHECK-functionality (part of the docker engine). This could help me detect and restart the container in case it doesn't function properly, according to its own provided API.

In the dev tag I have now a container image that includes a health check. @jeroenhendricksen Can you check if this also works in your environment?

jeroenhe commented 4 years ago

Not sure if it fits here, but what I'd like to have is the process offering a (HTTP API) health endpoint I can query using the HEALTHCHECK-functionality (part of the docker engine). This could help me detect and restart the container in case it doesn't function properly, according to its own provided API.

In the dev tag I have now a container image that includes a health check. @jeroenhendricksen Can you check if this also works in your environment?

It seems to work perfectly after only pulling the latest image and restarting it! The healthcheck reports a good health:

detecticam ./DetectiCam --configdir=/ ... Up (healthy) 80/tcp

I haven't really found a good way to make detecticam unhealthy (after first running a while), but when the container dies, it will auto restart itself thanks to autoheal (and yes, i need a kubernetes cluster ;-)

RaimondB commented 4 years ago

I haven't really found a good way to make detecticam unhealthy (after first running a while), but when the container dies, it will auto restart itself thanks to autoheal (and yes, i need a kubernetes cluster ;-)

I have tested the restarting by using docker stack deploy. And converting my node to a docker swarm single node cluster. K8s is just a bit to much for the home automation I have experienced :-) . But swarm seems to be a good alternative. I am now trying out portainer as a management console for the swarm. https://www.portainer.io/documentation/ . One drawback is that the containernames now become very long. And this messes up the docker grafana dashboard I have. But maybe I'll try out the Elastic + Prometheus setup. I have seen some nice Grafana dashboards for that stack.

jeroenhe commented 4 years ago

I have tested the restarting by using docker stack deploy. And converting my node to a docker swarm single node cluster. K8s is just a bit to much for the home automation I have experienced :-) . But swarm seems to be a good alternative. I am now trying out portainer as a management console for the swarm. https://www.portainer.io/documentation/ . One drawback is that the containernames now become very long. And this messes up the docker grafana dashboard I have. But maybe I'll try out the Elastic + Prometheus setup. I have seen some nice Grafana dashboards for that stack.

You should consider giving Rancher's k3s a try. It's an easy to install, lightweight Kubernetes. Only reason I'm not yet running in yet is that when I install it, the traefik ingress controller that comes along with it hijacks my 443 port and all of my services are no longer accessible, so I can't easily migrate to k3s.

I don't know if this applies when you rely onDocker swarm stack files, but container names (for the underlying docker containers) can be controlled when using docker-compose. You can provide the container_name inside the docker-compose.yml per container, which allows you to control your containers with the container name that you choose.

RaimondB commented 4 years ago

Indeed I was also looking at K3s. But deploying with Kubernetes is much more involved than a docker-compose file. I have done some experiments with openhab on microK8s but it does not play nice with Ingress because of the way it redirects when you go to the root of openhab where it will show the dashboard.

The container_name indeed you can specify (and I do). But this is not used in swarm mode. It will use the name of the stack you choose to deploy with and then postfix stuff to uniquely identify replicas for when you are using a real cluster, doing rolling upgrades etc...