Investigate PromScale database setup

[nascentcore-eng]

Goal is to better understand how promscale works, and how it can be used to support Starship open source and commercial product.

Right now when using psql connect to timescale db, and connect to postgres database

kubectl exec -it timescaledb-0 -- sh
# Then in the shell running in the container, start psql
psql
\c postgres
\dt+

\dt+ above shows no data tables.

This is confusing. As we know that prom and otel data are sent to promscale, which then writes into timescaledb, according to promscale documentation in https://docs.timescale.com/promscale/latest/about-promscale/

The goal here is to investigate and understand what's underlying mechanism used by promscale and timescale DB for storing prom and otel data, and how they serve the queries.

[JaredTan95]

following promescale official docs:

• https://docs.timescale.com/promscale/latest/query-data/query-traces/
• https://docs.timescale.com/promscale/latest/query-data/query-traces/#views

Promscale uses views to provide a stable interface. The set of views is provided in the ps_trace schema.

kubectl exec -it timescaledb-0 -- sh
# Then in the shell running in the container, start psql
psql
\c postgres
postgres=# \dn
           List of schemas
           Name           |  Owner
--------------------------+----------
 _prom_catalog            | postgres
 _prom_ext                | postgres
 _ps_catalog              | postgres
 _ps_trace                | postgres
 _timescaledb_cache       | postgres
 _timescaledb_catalog     | postgres
 _timescaledb_config      | postgres
 _timescaledb_internal    | postgres
 prom_api                 | postgres
 prom_data                | postgres
 prom_data_exemplar       | postgres
 prom_data_series         | postgres
 prom_info                | postgres
 prom_metric              | postgres
 prom_series              | postgres
 ps_tag                   | postgres
 ps_trace                 | postgres
 public                   | postgres
 timescaledb_experimental | postgres
 timescaledb_information  | postgres
(20 rows)

then query data with sql by view:

postgres=# SELECT * from go_gc_duration_seconds
WHERE time > now() - INTERVAL '5 minutes';

[nascentcore-eng]

https://www.timescale.com/blog/promscale-0-4-drawing-inspiration-from-cortex-to-rewrite-support-for-prometheus-high-availability/

Another interesting question is if and how promscale actually deduplicate prometheus data (as prometheus metrics have a lot of labels, without deduplication it would take too much space).

@JaredTan95 found that Prometheus agent mode actually disables local compaction of metrics data, so it means some system later in the pipeline should perform compaction, or the data will explode.

The provided link is about promscale itself.

[nascentcore-eng]

Unassign @JaredTan95 as we should focus on pre-0.1 public release

[nascentcore-eng]

A followup question: Right now we have 3 instances of promscale pod Do we need that much? Or just 1 instance is sufficient, assuming that 3 instances are stateless replicas. If the multiple instances are coordinating, then it's a different store.

[nascentcore-eng]

Another question/topic: https://docs.timescale.com/promscale/latest/scale-ha/high-availability/ HA configuration for promscale, which means we need to figure out the configuration for promscale to be highly-available. This is in part related to https://github.com/tricorder-observability/starship/issues/430 multi node postgres

[nascentcore-eng]

PromScale is deprecated No need to continue this...