timescale / tobs

tobs - The Observability Stack for Kubernetes. Easy install of a full observability stack into a k8s cluster with Helm charts.
Apache License 2.0
561 stars 61 forks source link

Promscale fails to start sighting version mismatch #505

Closed nhudson closed 2 years ago

nhudson commented 2 years ago

What happened?

When running make helm-install Promscale will show errors where an extention version mismatch is present and will not start properly.

How to reproduce it (as minimally and precisely as possible):

Run make helm-install locally. You will need Docker and Kind installed

2022/07/25 20:52:30 maxprocs: Leaving GOMAXPROCS=6: CPU quota undefined
level=info ts=2022-07-25T20:52:30.805Z caller=runner.go:75 msg="Version: 0.13.0, Commit Hash: e7745aa1b95ac4d58f18beef9d06924062f5ee4c, Branch: HEAD"
level=info ts=2022-07-25T20:52:30.805Z caller=runner.go:80 config="{ListenAddr::9201 ThanosStoreAPIListenAddr: OTLPGRPCListenAddr::9202 PgmodelCfg:{CacheConfig:{SeriesCacheInitialSize:250000 seriesCacheMemoryMaxFlag:{kind:0 value:50} SeriesCacheMemoryMaxBytes:4167720960 MetricsCacheSize:10000 LabelsCacheSize:100000 ExemplarKeyPosCacheSize:10000 InvertedLabelsCacheSize:500000} AppName:promscale@0.13.0 Host:test.default.svc Port:5432 User:postgres Password:**** Database:postgres SslMode:require DbConnectionTimeout:1m0s IgnoreCompressedChunks:false AsyncAcks:false WriteConnections:0 WriterPoolSize:-1 ReaderPoolSize:-1 MaxConnections:-1 UsesHA:true DbUri:**** EnableStatementsCache:true} LogCfg:{Level:info Format:logfmt} TracerCfg:{OtelCollectorEndpoint: OtelCollectorTLSCertFile: OtelCollectorTLSKeyFile: JaegerCollectorEndpoint: SamplingRatio:1} APICfg:{AllowedOrigin:^(?:.*)$ ReadOnly:false HighAvailability:true AdminAPIEnabled:false TelemetryPath:/metrics Auth:0x40006474a0 MultiTenancy:<nil> Rules:<nil>} LimitsCfg:{targetMemoryFlag:{kind:0 value:80} TargetMemoryBytes:8335441920} TenancyCfg:{SkipTenantValidation:false EnableMultiTenancy:false AllowNonMTWrites:false ValidTenantsStr:allow-all ValidTenantsList:[]} PromQLCfg:{EnabledFeatureMap:map[] PromscaleEnabledFeatureList:[] MaxQueryTimeout:2m0s SubQueryStepInterval:1m0s LookBackDelta:5m0s MaxSamples:50000000 MaxPointsPerTs:11000} RulesCfg:{NotificationQueueCapacity:10000 OutageTolerance:1h0m0s ForGracePeriod:10m0s ResendDelay:1m0s PrometheusConfigAddress: PrometheusConfig:global:\n  scrape_interval: 1m\n  scrape_timeout: 10s\n  evaluation_interval: 1m\n} TracingCfg:{MaxTraceDuration:1h0m0s} ConfigFile:/etc/promscale/config.yaml DatasetConfig:metrics:\n  compress_data: true\n  default_retention_period: 90d\ntraces:\n  default_retention_period: 30d\n TLSCertFile: TLSKeyFile: ThroughputInterval:1s Migrate:true StopAfterMigrate:false UseVersionLease:true InstallExtensions:true UpgradeExtensions:true UpgradePrereleaseExtensions:false StartupOnly:false}"
level=warn ts=2022-07-25T20:52:31.295Z caller=extension.go:389 msg="skipping upgrade to prerelease version 0.1.3-beta, use -startup.upgrade-prerelease-extensions to force."
level=warn ts=2022-07-25T20:52:31.298Z caller=new_migrate.go:129 msg="as of Promscale version 0.11.0, the Promscale database extension is mandatory. please install the latest version of this extension in your TimescaleDB/PostgreSQL database in order to proceed."
level=error ts=2022-07-25T20:52:31.299Z caller=runner.go:116 msg="aborting startup due to error" err="migration error: error while checking for supported and available promscale extension versions: The promscale extension is not available at the right version, need version: >=0.5.4 <0.5.99 and the default version is 0.5.0 "
nhudson commented 2 years ago

This is most likely an unrelated issue, but just to be safe I am noting it here. https://github.com/timescale/promscale/issues/1520

nhudson commented 2 years ago

Looks like its an incompatability issue between the promscale 0.13.0 Helm chart and the timescaledb-single 0.14.0 Helm chart. timescaledb-single has version pg14.3-ts2.7.0-p0 pinned, which we also track in the tobs values.yaml here.

Forcing an update of timescaleb to pg14.4-ts2.7.2-p0 fixes this issue.