search

shatteredsilicon / ssm-submodules

GNU Affero General Public License v3.0
1 stars 2 forks source link

In-place upgrade (most likely with multiple rds/remote instances) of SSM server causes qan config missing/changed #164

Closed oblitorum closed 1 year ago

oblitorum commented 1 year ago

After upgrading .1 to .3, some of the qan configs are missing and it causes ssm-managed failed to start.

time="2023-08-23T18:24:11Z" level=error msg="getQANConfig response:\n\nHTTP/1.1 502 Bad Gateway\r\nContent-Length: 166\r\nConnection: keep-alive\r\nContent-Type: text/html\r\nDate: Wed, 23 Aug 2023 18:24:11 GMT\r\nServer: nginx\r\n\r\n<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n\n" component=qan request=main
time="2023-08-23T18:24:11Z" level=info msg="restoreConfigs err: failed to get qan config from QAN API: unexpected QAN response status code 502" component=qan request=main
time="2023-08-23T18:24:11Z" level=panic msg="RDS service problem: unexpected QAN response status code 502\ngithub.com/shatteredsilicon/ssm-managed/services/qan.(*Service).getQANConfig\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/qan/qan.go:854\ngithub.com/shatteredsilicon/ssm-managed/services/qan.(*Service).restoreConfigs\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/qan/qan.go:363\ngithub.com/shatteredsilicon/ssm-managed/services/qan.(*Service).Restore\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/qan/qan.go:181\ngithub.com/shatteredsilicon/ssm-managed/services/rds.(*Service).Restore\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/rds/rds.go:1010\nmain.makeRDSService.func2\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:201\ngithub.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform%2ev1.(*DB).InTransactionContext\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform.v1/db.go:93\ngithub.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform%2ev1.(*DB).InTransaction\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform.v1/db.go:74\nmain.makeRDSService\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:200\nmain.main\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:617\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594\nfailed to get qan config from QAN API\ngithub.com/shatteredsilicon/ssm-managed/services/qan.(*Service).restoreConfigs\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/qan/qan.go:365\ngithub.com/shatteredsilicon/ssm-managed/services/qan.(*Service).Restore\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/qan/qan.go:181\ngithub.com/shatteredsilicon/ssm-managed/services/rds.(*Service).Restore\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/services/rds/rds.go:1010\nmain.makeRDSService.func2\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:201\ngithub.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform%2ev1.(*DB).InTransactionContext\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform.v1/db.go:93\ngithub.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform%2ev1.(*DB).InTransaction\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/gopkg.in/reform.v1/db.go:74\nmain.makeRDSService\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:200\nmain.main\n\t/builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:617\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594" component=main
time="2023-08-23T18:24:11Z" level=info msg=Done. component=main
panic: (*logrus.Entry) 0xc000222d90

goroutine 1 [running]:
github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus.(*Entry).log(0xc00051a700, 0x0, {0xc000665800, 0xbb5})
    /builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus/entry.go:259 +0x45b
github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus.(*Entry).Log(0xc00051a700, 0x0, {0xc000331cb0?, 0x1?, 0x1?})
    /builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus/entry.go:293 +0x4f
github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus.(*Entry).Logf(0xc00051a700, 0x0, {0xec9e03?, 0x6c?}, {0xc000331e70?, 0x8?, 0xc00058ff00?})
    /builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus/entry.go:338 +0x85
github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus.(*Entry).Panicf(...)
    /builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/vendor/github.com/sirupsen/logrus/entry.go:376
main.main()
    /builddir/build/BUILD/go/src/github.com/shatteredsilicon/ssm-managed/cmd/ssm-managed/main.go:619 +0xc05
ssm-managed 8.8.1.17.5.3-1

After upgrading .2 to .3, the qan config Collect from of some RDS instances changed to performance schema.

image

@gordan-bobic

gordan-bobic commented 1 year ago

Thanks for figuring this out. Please fix for next release. We'll avoid upgrading until then.