search

rudderlabs / rudder-server

Privacy and Security focused Segment-alternative, in Golang and React
https://www.rudderstack.com/
Other
4.04k stars 311 forks source link

backupDSLoop exception: mark end of backup operation: driver: bad connection; driver: bad connection #4535

Open kalavt opened 4 months ago

kalavt commented 4 months ago

Describe the bug mark end of backup operation: driver: bad connection; driver: bad connection

Expected behavior version: 1.22.1

Screenshots ` 2024-04-02T13:32:11.775Z ERROR utils.misc misc/misc.go:926 Panic detected. Application will crash. {"stack": "goroutine 430 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1()\n\t/rudder-server/utils/misc/misc.go:927 +0x37e\nsync.(Once).doSlow(0x60f1280?, 0xc002b31e18?)\n\t/usr/local/go/src/sync/once.go:74 +0xbf\nsync.(Once).Do(...)\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1()\n\t/rudder-server/utils/misc/misc.go:916 +0xbb\npanic({0x32a8f60?, 0xc08b668dc0?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x21f\ngithub.com/rudderlabs/rudder-server/jobsdb.(Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860})\n\t/rudder-server/jobsdb/backup.go:105 +0x345\ngithub.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.func1()\n\t/rudder-server/jobsdb/jobsdb.go:1016 +0x1f\ngithub.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.WithBugsnag.func2()\n\t/rudder-server/utils/misc/misc.go:945 +0x5c\ngolang.org/x/sync/errgroup.(Group).Go.func1()\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(Group).Go in goroutine 573\n\t/go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96\n", "panic": "mark end of backup operation: driver: bad connection; driver: bad connection", "team": "Core", "goRoutines": 1169, "version": "1.22.1", "releaseStage": "development"} 2024-04-02T13:32:11.776Z ERROR utils.misc misc/misc.go:926 goroutine 430 [running]: github.com/rudderlabs/rudder-go-kit/logger.(logger).Fatalw(0xc001490a50, {0x38fbe93?, 0x4bf?}, {0xc089ae1500?, 0x1?, 0x4ffbcc?}) /go/pkg/mod/github.com/rudderlabs/rudder-go-kit@v0.23.2/logger/logger.go:332 +0xa5 github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1() /rudder-server/utils/misc/misc.go:926 +0x5f6 sync.(Once).doSlow(0x60f1280?, 0xc002b31e18?) /usr/local/go/src/sync/once.go:74 +0xbf sync.(Once).Do(...) /usr/local/go/src/sync/once.go:65 github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1() /rudder-server/utils/misc/misc.go:916 +0xbb panic({0x32a8f60?, 0xc08b668dc0?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/rudderlabs/rudder-server/jobsdb.(Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860}) /rudder-server/jobsdb/backup.go:105 +0x345 github.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.func1() /rudder-server/jobsdb/jobsdb.go:1016 +0x1f github.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.WithBugsnag.func2() /rudder-server/utils/misc/misc.go:945 +0x5c golang.org/x/sync/errgroup.(Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573 /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

2024/04/02 13:32:11 notifying bugsnag: mark end of backup operation: driver: bad connection; driver: bad connection 2024/04/02 13:32:11 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: '' 2024/04/02 13:32:11 bugsnag/sessions/publisher.publish invalid API key: '' panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered] panic: mark end of backup operation: driver: bad connection; driver: bad connection [recovered] panic: mark end of backup operation: driver: bad connection; driver: bad connection

goroutine 430 [running]: github.com/bugsnag/bugsnag-go/v2.AutoNotify({0xc06e6099f0, 0x3, 0x7f09e72068b8?}) /go/pkg/mod/github.com/bugsnag/bugsnag-go/v2@v2.3.0/bugsnag.go:114 +0x3e8 panic({0x32a8f60?, 0xc08b668dc0?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1.1() /rudder-server/utils/misc/misc.go:935 +0x614 sync.(Once).doSlow(0x60f1280?, 0xc002b31e18?) /usr/local/go/src/sync/once.go:74 +0xbf sync.(Once).Do(...) /usr/local/go/src/sync/once.go:65 github.com/rudderlabs/rudder-server/utils/misc.BugsnagNotify.func1() /rudder-server/utils/misc/misc.go:916 +0xbb panic({0x32a8f60?, 0xc08b668dc0?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/rudderlabs/rudder-server/jobsdb.(Handle).backupDSLoop(0xc00147f880, {0x40ae8d0, 0xc00086d860}) /rudder-server/jobsdb/backup.go:105 +0x345 github.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.func1() /rudder-server/jobsdb/jobsdb.go:1016 +0x1f github.com/rudderlabs/rudder-server/jobsdb.(Handle).startBackupDSLoop.WithBugsnag.func2() /rudder-server/utils/misc/misc.go:945 +0x5c golang.org/x/sync/errgroup.(Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56 created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 573 /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96 `

gitcommitshow commented 4 months ago

@kalavt can you please share more details? How to reproduce this issue?

Digital365Staking commented 1 month ago

The error message you're seeing indicates that there's a problem with the database connection during the backup operation in the rudder-server application. The error message "driver: bad connection" typically means that the application is unable to establish a connection to the database⁴.

Here are a few steps you can take to troubleshoot this issue:

  1. Check the Database Connection: Ensure that the database is running and that the application has the correct connection details. If the database is running on a different server, check that the network connection between the two servers is working correctly.

  2. Check the Database Driver: The error message suggests there might be an issue with the database driver. Make sure you're using the correct driver for your database and that the driver is up-to-date.

  3. Inspect the Application Logs: The logs can provide more details about what the application was doing when the error occurred. Look for any error messages or warnings that occurred around the same time as the "driver: bad connection" error.

  4. Update Rudder Server: The error logs indicate that you're using Rudder Server version 1.22.1. If there's a newer version available, consider updating to see if the issue has been resolved in a later release.

Regarding the "invalid api key" error for Bugsnag, it seems like the API key provided to Bugsnag is not valid. You might want to check if the API key is correctly set in your configuration¹. If you're using different environments (like development and production), make sure to use the correct API key for each environment¹.

If none of these steps resolve the issue, you might want to consider reaching out to the Rudder Server community or support for more specific guidance. They might be able to provide more insight based on the specifics of your setup and the exact error messages you're seeing²⁵.

(1) Troubleshooting :: Rudder Documentation. https://docs.rudder.io/reference/8.1/administration/troubleshooting.html. (2) javascript - Bugsnag: Invalid API key 'undefined' - Stack Overflow. https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined. (3) Docker setup is broken · Issue #448 · rudderlabs/rudder-server. https://github.com/rudderlabs/rudder-server/issues/448. (4) backupDSLoop exception: mark end of backup operation: driver: bad .... https://github.com/rudderlabs/rudder-server/issues/4535. (5) Bugsnag | RudderStack Docs. https://www.rudderstack.com/docs/destinations/streaming-destinations/bugsnag/. (6) undefined. https://docs.rudderstack.com/get-started/installing-and-setting-up-rudderstack/docker. (7) undefined. https://github.com/segment-boneyard/analytics.js-integrations/issues/61. (8) undefined. https://d[2](https://stackoverflow.com/questions/42693851/bugsnag-invalid-api-key-undefined)wy8f7a9ursnm.cloudfront.net/bugsnag-3.min.js.