"Figure 13 shows the components that were added to the project architecture so as to
provide the monitoring dashboard described above. First, we set up a simple Extract-
Transform-Load (ETL) process in Python (second box of the bottom row), which
fetches the API logs periodically and transforms them into partitioned Parquet files.
Then, we use the Quarto framework21 to build an interactive dashboard (third box
of the bottom row). To compute the various metrics presented in the dashboard, the
Parquet files are queried using the SQL language through the DuckDB engine. Like
the API, the dashboard is built and deployed as a container on the Kubernetes cluster,
and this process is automated using a CI/CD pipeline. The annotation component
(fourth box of the bottom row) is discussed in the next section."
I think we can add a note that this is a bit manual and custom made. But we are open to look for more standard tools
"Figure 13 shows the components that were added to the project architecture so as to provide the monitoring dashboard described above. First, we set up a simple Extract- Transform-Load (ETL) process in Python (second box of the bottom row), which fetches the API logs periodically and transforms them into partitioned Parquet files. Then, we use the Quarto framework21 to build an interactive dashboard (third box of the bottom row). To compute the various metrics presented in the dashboard, the Parquet files are queried using the SQL language through the DuckDB engine. Like the API, the dashboard is built and deployed as a container on the Kubernetes cluster, and this process is automated using a CI/CD pipeline. The annotation component (fourth box of the bottom row) is discussed in the next section."
I think we can add a note that this is a bit manual and custom made. But we are open to look for more standard tools