Demo application - Live dashboard for logs/inferences

[robmarkcole]

One application I am interested in developing is a live dashboard that could show historical inferences on Deepstack. This could probably use Grafana for frontend and Postgres/influxdb + S3/minio for backend. The question is how to integrate with Deepstack? I thought about creating an app which proxies deepstack, i.e. redirect requests and captures all the image and inference data. Another idea would be to somehow integrate with the Deepstack logging, perhaps via Redis. Interested to get feedback and suggestions on this issue.

from Intel

[LucidityCrash]

What about Prometheus for the metrics ... you could either use the textfile collector in node_exporter and have deepstack send its logs in the right format into an appropriate folder, or you could write a metrics endpoint into the app, or you could have some generic api for fetching metrics and then have a small go deepstack_exporter to fetch these.

Not sure how you'd do the video streaming bit though

[robmarkcole]

Prometheus is basically a db? Can it be brought up via a docker compose? Once we have decided to work on this we can evaluate the tradeoffs of the different options

[LucidityCrash]

Prometheus is a time series db, quite often used in a docker container and for the kind of data you are showing on your mockup it would be ideal ... There are others eg influx but I use this at work and at home, config and architecture is simple ... it scrapes a metric endpoint which can be running anywhere which returns the metrics you are gathering as a list of key value pairs one per line. How you go about generating that is pretty much up to you, go is a very common for exporters. If you have a massive infrastructure or a need to keep decades of metrics you can extend its capabilities using something called Thanatos but I can't see the need for that here :).

[johnolafenwa]

Hello @LucidityCrash , thank you for the suggestion. Prometheus is definitely a perfect fit for this. Our plan is to incorporate a /metrics endpoint which external tools like prometheus can poll for metric data. DeepStack currently logs to stdout and stderr only and to files located in /logs in the container. However, the metric and healthpoints we are going to add will be the primary way to integrate external tools to monitor deepstack.

[robmarkcole]

Note to self to also checkout Redis time series, I remember seeing a nice integration with an object detection system

https://redislabs.com/blog/my-other-stack-is-redisedge/

[robmarkcole]

My google searches keep bringing me back to https://datasette.io/ which allows building a data explorer app with sqlite backend. A variety of plugins add useful functionality such as previewing thumbnails and allowing displaying simple maps

[robmarkcole]

I think this could be as simple as having a process which is monitoring the redis queue, and doing something like https://redislabs.com/blog/how-to-manage-real-time-iot-sensor-data-in-redis/