Bug: InitializeTemplates failed error on HA cluster install

[chenilim]

Steps to reproduce the behavior

1. Create a new HA test cluster (via /cloud create SERVERNAME --size miniHA --database mysql-operator --version master)
2. Wait for the server to be provisioned, then log in
3. Open the server logs in the System Console
4. See the following error: Error 1062: Duplicate entry '0-3fa520eb-30cd-4852-829a-ba3bc7e88e26' for key 'PRIMARY'.

Expected behavior

No error

Edition and Platform

• Edition: Mattermost plugin
• Version: v0.9.2

Additional context

Note: Despite the error, the Boards plugin appears to be running on both nodes of the cluster, and functions.

The block id in the error is the first block in the InitializeTemplates method. It seems that one node of the (2-node) cluster successfully runs InitializeTemplates, but this might be happening before isInitializationNeeded runs on the second one, which then tries to importInitialTemplates, and fails because the blocks are already inserted.

Perhaps one solution is to wrap isInitializationNeeded and importInitialTemplates in a single transaction?

More lines from the log:

{"timestamp":"2021-10-11 16:14:35.258 Z","level":"info","msg":"debug [2021-10-11 16:14:35.258 Z] importInitialTemplates caller=\"mlog/mlog.go:213\"\n","caller":"io/io.go:425","plugin_id":"focalboard","source":"plugin_stdout"}
{"timestamp":"2021-10-11 16:14:35.259 Z","level":"info","msg":"debug [2021-10-11 16:14:35.259 Z] Inserting blocks caller=\"mlog/mlog.go:213\" block_count=36\n","caller":"io/io.go:425","plugin_id":"focalboard","source":"plugin_stdout"}
{"timestamp":"2021-10-11 16:14:35.293 Z","level":"info","msg":"error [2021-10-11 16:14:35.293 Z] InitializeTemplates failed caller=\"mlog/mlog.go:228\" error=\"Error 1062: Duplicate entry '0-3fa520eb-30cd-4852-829a-ba3bc7e88e26' for key 'PRIMARY'\"\n","caller":"io/io.go:425","plugin_id":"focalboard","source":"plugin_stdout"}
{"timestamp":"2021-10-11 16:14:35.306 Z","level":"error","msg":"Failed to install prepackaged plugin","caller":"app/plugin.go:887","path":"/mattermost/prepackaged_plugins/focalboard-v0.9.2-linux-amd64.tar.gz","error":"Failed to install extracted prepackaged plugin /mattermost/prepackaged_plugins/focalboard-v0.9.2-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., error initializing the DB: Error 1062: Duplicate entry '0-3fa520eb-30cd-4852-829a-ba3bc7e88e26' for key 'PRIMARY'"}
{"timestamp":"2021-10-11 16:14:39.659 Z","level":"info","msg":"Metrics endpoint is initiated","caller":"metrics/metrics.go:882","address":":8067"}

[mgdelacroix]

@chenilim could not reproduce the error, although your analysis makes sense. There is a potential issue here, as running into this error would cause the plugin to fail on start. If the errored plugin node is restarted everything would work fine, hence the lack of visible consequences of this.

What we could do to mitigate this is to avoid failing if the templates are not inserted, so the second node that tries to insert them simply logs the error and continues with the boot process.