timescale / tobs

tobs - The Observability Stack for Kubernetes. Easy install of a full observability stack into a k8s cluster with Helm charts.
Apache License 2.0
563 stars 60 forks source link

Setup connection to external TimescaleDB #531

Closed nhudson closed 2 years ago

nhudson commented 2 years ago

Description

Create a way to connect to and setup Grafana to query an external TimescaleDB if it's not provisioned by tobs.

You can now easily install tobs to connect to an external (to tobs) TimescaleDB. This can be achieved by setting timescaledb-single.enabled=false, setting the password for the TimescaleDB postgres role in promscale.connection.password, and the host URI for timescale in promscale.connection.host.

This at minimum will setup Promscale and the Grafana datasource connections correctly to the external TimescaleDB.

Type of change

What type of changes does your code introduce to tobs? Put an x in the box that apply.

nhudson commented 2 years ago

I wanted to be able to write a new ct test for this feature, but it would seem that there are artifacts left behind when helm deletes the installation. See #365 for more information. I don't think we could add more tests until we can work around this issue.

nhudson commented 2 years ago

WDYT about using promscale.connection.uri in this test instead of dbName, user, password, host combo? This way we could tackle #331 at the same time.

Yeah I was thinking of looking into seeing how we could use promscale.connection.uri after looking at #536

nhudson commented 2 years ago

e2e tests are failing, but seem to be timing out when trying to setup the 2nd ct test that was added to this PR.

Installing chart with values file 'chart/ci/externaldb-values.yaml'...

Creating namespace 'chart-fzd15xu19p'...
namespace/chart-fzd15xu19p created
Error: The operation was canceled.
paulfantom commented 2 years ago

Seems like the issue is with insufficient amount of resources:

2022-08-16T17:48:13.4171151Z   Warning  FailedScheduling  4m36s (x5 over 15m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Maybe for this particular test let's provision only one replica of prometheus and one replica of alertmanager by specifying it in chart/ci/externaldb-values.yaml

nhudson commented 2 years ago

Only merge after #546

nhudson commented 2 years ago

Seems like the issue is with insufficient amount of resources:

2022-08-16T17:48:13.4171151Z   Warning  FailedScheduling  4m36s (x5 over 15m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Maybe for this particular test let's provision only one replica of prometheus and one replica of alertmanager by specifying it in chart/ci/externaldb-values.yaml

@paulfantom I am interested how you saw this error? Also looks like that doesn't work either, this time the job has been running for hours now.

paulfantom commented 2 years ago

I opened raw logs from one of the CI runs and it was there. However right now it is no longer present.

nhudson commented 2 years ago

DO NOT merge until after #555