search

mattermost / mattermost-helm

Mattermost Helm charts for Kubernetes
Apache License 2.0
162 stars 145 forks source link

Mattermost-team-edition does not startup on some clusters #411

Open mustdiechik opened 12 months ago

mustdiechik commented 12 months ago

test environment:

I patched the chart and add additional loadBalancer for calls:

...
  loadBalancerIP: x.x.x.x
  externalTrafficPolicy: Local
  ports:
  - name: rtc
    port: 8443
    protocol: UDP
  type: LoadBalancer
...

The same configuration with only different loadBalancerIP tested on 4 k8s clusters. One of them - minikube, 3 other - managed by cloud provider (test, stage, prod).

I have strange problem with prod cluster - in 3 other all works as expected. Pod didn`t switch in Running state. 7.10.3 logs:

mattermost@mattermost-5ddcdf4665-tzkcw:~$ cat /mattermost/logs/mattermost.log 
{"timestamp":"2023-07-17 22:56:20.669 Z","level":"info","msg":"Server is initializing...","caller":"platform/service.go:165","go_version":"go1.19.5"}
{"timestamp":"2023-07-17 22:56:20.669 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:242","database":"master","dataSource":"postgres:__REDACTED__?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 22:56:20.993 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"info","msg":"Starting websocket hubs","caller":"platform/web_hub.go:95","number_of_hubs":16}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":1}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":3}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":2}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":0}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":6}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":7}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":9}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":4}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":8}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":10}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":11}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":12}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":13}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":5}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":14}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":15}
{"timestamp":"2023-07-17 22:56:21.066 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:132","for locale":"en","from locale":"/mattermost/i18n/en.json"}
{"timestamp":"2023-07-17 22:56:21.072 Z","level":"info","msg":"Ensuring the telemetry ID","caller":"telemetry/telemetry.go:142","id":"nqpe5ooaujypxqmjh56ucxsmbe"}
{"timestamp":"2023-07-17 22:56:21.132 Z","level":"debug","msg":"Email batching job starting. Checking for pending emails periodically.","caller":"email/email_batching.go:90","interval_in_seconds":30}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Current version is 7.10.3 (7.10.3/Wed Jun 14 12:09:40 UTC 2023/74af1505a69a560085fd609620e3f92123e84536/none)","caller":"app/server.go:412","current_version":"7.10.3","build_number":"7.10.3","build_date":"Wed Jun 14 12:09:40 UTC 2023","build_hash":"74af1505a69a560085fd609620e3f92123e84536","build_hash_enterprise":"none"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Team Edition Build","caller":"app/server.go:423","enterprise_build":false}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Printing current working","caller":"app/server.go:427","directory":"/mattermost"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"info","msg":"Loaded config","caller":"app/server.go:428","source":"postgres://%2A%2A%2A%2A:%2A%2A%2A%2A@10.92.33.52:6432/mattermost?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 22:56:21.251 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"info","msg":"Starting workers","caller":"jobs/workers.go:48"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Cleaning up command webhook store.","caller":"sqlstore/command_webhook_store.go:93"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"scheduler/worker.go:39","worker":"Plugins"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Watcher Started","caller":"jobs/jobs_watcher.go:38"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker Started","caller":"indexer/indexing_job.go:100","workername":"BleveIndexer"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportDelete"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"resend_invitation_email/worker.go:54","worker":"resend_invitation_email"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportProcess"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExtractContent"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ProductNotices"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"HostedPurchaseScreening"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportProcess"}
{"timestamp":"2023-07-17 22:56:21.321 Z","level":"debug","msg":"Worker started","caller":"migrations/worker.go:50","worker":"Migrations"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"InstallNotifyAdmin"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Cleaning up session store.","caller":"app/server.go:1277"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportDelete"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExpiryNotify"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"info","msg":"Starting schedulers.","caller":"jobs/schedulers.go:47"}
{"timestamp":"2023-07-17 22:56:21.322 Z","level":"debug","msg":"Cleaning up token store.","caller":"app/server.go:1262"}
{"timestamp":"2023-07-17 22:56:21.437 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:104","filename":"/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2023-07-17 22:56:21.446 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"plugins","next_runtime":"2023-07-18 22:56:22.336425601 +0000 UTC m=+86402.403838454"}
{"timestamp":"2023-07-17 22:56:21.472 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:101","error":"resource: License id: "}
{"timestamp":"2023-07-17 22:56:21.473 Z","level":"debug","msg":"Skipping license expiration check because no license is expected on Team Edition","caller":"app/server.go:1379"}
{"timestamp":"2023-07-17 22:56:21.525 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"expiry_notify","next_runtime":"2023-07-17 23:06:22.754042035 +0000 UTC m=+602.821454887"}
{"timestamp":"2023-07-17 22:56:21.590 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"product_notices","next_runtime":"2023-07-17 23:56:21.590547182 +0000 UTC m=+3601.657960032"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"export_delete","next_runtime":"2023-07-18 22:56:22.184000011 +0000 UTC m=+86402.251412877"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_post/scheduler.go:20","scheduler":"last_accessible_post"}
{"timestamp":"2023-07-17 22:56:21.653 Z","level":"debug","msg":"Scheduler: isEnabled: true","caller":"notify_admin/install_plugin_scheduler.go:20","scheduler":"install_plugin_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.715 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"install_plugin_notify_admin","next_runtime":"2023-07-18 22:56:22.559660692 +0000 UTC m=+86402.627073549"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"2023-07-17 22:57:21.775861481 +0000 UTC m=+61.843274332"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_file/scheduler.go:20","scheduler":"last_accessible_file"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"upgrade_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.775 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"trial_notify_admin"}
{"timestamp":"2023-07-17 22:56:21.837 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"import_delete","next_runtime":"2023-07-18 22:56:23.515543916 +0000 UTC m=+86403.582956768"}
{"timestamp":"2023-07-17 22:56:22.736 Z","level":"info","msg":"Skipping triggering Elasticsearch channel index fix job as build is not Enterprise ready","caller":"app/migrations.go:570"}
{"timestamp":"2023-07-17 22:56:22.752 Z","level":"debug","msg":"Initializing web routes","caller":"web/web.go:26"}
{"timestamp":"2023-07-17 22:56:22.753 Z","level":"debug","msg":"Rewriting static assets","caller":"utils/subpath.go:83","from_subpath":"/","to_subpath":"/"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Using client directory","caller":"web/static.go:34","clientDir":"/mattermost/client"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:218"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:613","interval_s":"30s"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:299"}
{"timestamp":"2023-07-17 22:56:22.789 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.calls.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.manland.mattermost-plugin-gitlab.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.custom-attributes.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.apps.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.matterpoll.matterpoll.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/github.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jitsi.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jenkins.tar.gz"}
{"timestamp":"2023-07-17 22:56:22.874 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/mattermost-autolink.tar.gz"}
{"timestamp":"2023-07-17 22:56:25.195 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jenkins","version":"1.1.0"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64args[plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64pid33"}
{"timestamp":"2023-07-17 22:56:25.337 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:25.345 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"address/tmp/plugin554633014networkunixtimestamp2023-07-17T22:56:25.345Z"}
{"timestamp":"2023-07-17 22:56:25.345 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"2023/07/17 22:56:25 LoadPluginConfiguration API failed to unmarshal: json: cannot unmarshal string into Go struct field CustomAttribute.CustomAttributes.GroupIDs of type []string","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.custom-attributes"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 22:56:25.352 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 22:56:25.380 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64args[plugins/mattermost-autolink/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:25.381 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64pid41"}
{"timestamp":"2023-07-17 22:56:25.381 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:25.400 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"networkunixaddress/tmp/plugin3266541804timestamp2023-07-17T22:56:25.400Z"}
{"timestamp":"2023-07-17 22:56:25.400 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:25.404 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 22:56:25.404 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 22:56:31.101 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"github","version":"2.1.6"}
{"timestamp":"2023-07-17 22:56:31.122 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.moussetc.mattermost.plugin.giphy","version":"2.1.0"}
{"timestamp":"2023-07-17 22:56:31.387 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64args[plugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.388 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64pid53"}
{"timestamp":"2023-07-17 22:56:31.388 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.409 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:31.409 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"address/tmp/plugin1920211718networkunixtimestamp2023-07-17T22:56:31.408Z"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid65"}
{"timestamp":"2023-07-17 22:56:31.463 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.488 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:31.488 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin194508154networkunixtimestamp2023-07-17T22:56:31.488Z"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64args[plugins/jitsi/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64pid77"}
{"timestamp":"2023-07-17 22:56:31.792 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"warn","msg":"ignored duplicate step name","caller":"app/plugin_api.go:979","plugin_id":"com.github.manland.mattermost-plugin-gitlab","name":"oauth-info","flow":"oauth"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 22:56:31.796 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 22:56:31.800 Z","level":"debug","msg":"ensured bot","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","id":"rx9bx5d3ntrhumudpw3qf387ce","username":"appsbot","plugin_caller":"server/plugin.go:68"}
{"timestamp":"2023-07-17 22:56:31.814 Z","level":"debug","msg":"failed to fetch license twice. May incorrectly default to on-prem mode","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/config/service.go:226"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"configured the plugin.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","developer_mode":"false","allow_http_apps":"true","version":"1.2.0","commit":"44db542","plugin_caller":"server/plugin.go:92","build_date":"Fri 06 Jan 2023 09:59:54 PM UTC","cloud_mode":"false"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"created an AWS client","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","access":"<nil>","secret":"<nil>","plugin_caller":"upstream/upaws/client.go:121","purpose":"Manifest store","region":"us-east-1"}
{"timestamp":"2023-07-17 22:56:31.815 Z","level":"debug","msg":"initialized API and persistent store","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:106"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"HTTP","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"Skipped upstream: AWS Lambda: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"Mattermost Plugin","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 22:56:31.816 Z","level":"debug","msg":"Skipped upstream: OpenFaaS: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/proxy/service.go:227","error":"OPENFAAS_URL environment variable must be defined: not found"}
{"timestamp":"2023-07-17 22:56:31.817 Z","level":"debug","msg":"initialized the app proxy","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:122"}
{"timestamp":"2023-07-17 22:56:31.818 Z","level":"debug","msg":"initialized incoming HTTP","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:125"}
{"timestamp":"2023-07-17 22:56:31.818 Z","level":"info","msg":"activated","caller":"app/plugin_api.go:973","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:135"}
{"timestamp":"2023-07-17 22:56:31.819 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 22:56:31.819 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 22:56:32.084 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"address/tmp/plugin2141876054networkunixtimestamp2023-07-17T22:56:32.083Z"}
{"timestamp":"2023-07-17 22:56:32.084 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:32.207 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 22:56:32.207 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64args[plugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64pid90"}
{"timestamp":"2023-07-17 22:56:32.411 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:32.428 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"address/tmp/plugin3804533052networkunixtimestamp2023-07-17T22:56:32.428Z"}
{"timestamp":"2023-07-17 22:56:32.428 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:32.660 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 22:56:32.660 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 22:56:33.352 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 22:56:33.352 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid103"}
{"timestamp":"2023-07-17 22:56:33.353 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 22:56:33.376 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"address/tmp/plugin4227929444networkunixtimestamp2023-07-17T22:56:33.375Z"}
{"timestamp":"2023-07-17 22:56:33.376 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 22:56:33.396 Z","level":"debug","msg":"Initializing telemetry","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).initTelemetry telemetry.go:86"}
{"timestamp":"2023-07-17 22:56:33.398 Z","level":"debug","msg":"activating","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).OnActivate activate.go:44"}
{"timestamp":"2023-07-17 22:56:33.403 Z","level":"debug","msg":"cleaning up calls state","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).cleanUpState channel_state.go:251"}
{"timestamp":"2023-07-17 22:56:51.133 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}

After changed tag to 7.10.4 pod was up, but after delete\apply release it not swiching in Runnig state. Logs:

mattermost@mattermost-d89c98c89-qdxf4:~$ cat /mattermost/logs/mattermost.log 
{"timestamp":"2023-07-17 23:09:13.667 Z","level":"info","msg":"Server is initializing...","caller":"platform/service.go:165","go_version":"go1.19.5"}
{"timestamp":"2023-07-17 23:09:13.667 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:242","database":"master","dataSource":"__REDACTED__?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 23:09:13.991 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"info","msg":"Starting websocket hubs","caller":"platform/web_hub.go:95","number_of_hubs":16}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":2}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":1}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":3}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":0}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":6}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":4}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":5}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":8}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":9}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":7}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":10}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":14}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":12}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":13}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":15}
{"timestamp":"2023-07-17 23:09:14.065 Z","level":"debug","msg":"Hub is starting","caller":"platform/web_hub.go:373","index":11}
{"timestamp":"2023-07-17 23:09:14.066 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:132","for locale":"en","from locale":"/mattermost/i18n/en.json"}
{"timestamp":"2023-07-17 23:09:14.071 Z","level":"info","msg":"Ensuring the telemetry ID..","caller":"telemetry/telemetry.go:142"}
{"timestamp":"2023-07-17 23:09:14.139 Z","level":"info","msg":"telemetry ID is set","caller":"telemetry/telemetry.go:152","id":"i4j7i9w6rtnc5qiprkm7uxtucc"}
{"timestamp":"2023-07-17 23:09:14.140 Z","level":"debug","msg":"Email batching job starting. Checking for pending emails periodically.","caller":"email/email_batching.go:90","interval_in_seconds":30}
{"timestamp":"2023-07-17 23:09:14.258 Z","level":"info","msg":"Current version is 7.10.4 (7.10.4/Wed Jul  5 09:25:31 UTC 2023/f4bc16bc8d7890dad21cae3073663d6911320a96/none)","caller":"app/server.go:412","current_version":"7.10.4","build_number":"7.10.4","build_date":"Wed Jul  5 09:25:31 UTC 2023","build_hash":"f4bc16bc8d7890dad21cae3073663d6911320a96","build_hash_enterprise":"none"}
{"timestamp":"2023-07-17 23:09:14.258 Z","level":"info","msg":"Team Edition Build","caller":"app/server.go:423","enterprise_build":false}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"info","msg":"Printing current working","caller":"app/server.go:427","directory":"/mattermost"}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"info","msg":"Loaded config","caller":"app/server.go:428","source":"postgres://%2A%2A%2A%2A:%2A%2A%2A%2A@10.92.33.52:6432/mattermost?connect_timeout=3&sslmode=disable"}
{"timestamp":"2023-07-17 23:09:14.259 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"info","msg":"Starting workers","caller":"jobs/workers.go:48"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Cleaning up token store.","caller":"app/server.go:1262"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Cleaning up command webhook store.","caller":"sqlstore/command_webhook_store.go:93"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker Started","caller":"indexer/indexing_job.go:100","workername":"BleveIndexer"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ProductNotices"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"InstallNotifyAdmin"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"resend_invitation_email/worker.go:54","worker":"resend_invitation_email"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"HostedPurchaseScreening"}
{"timestamp":"2023-07-17 23:09:14.324 Z","level":"debug","msg":"Worker started","caller":"migrations/worker.go:50","worker":"Migrations"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"scheduler/worker.go:39","worker":"Plugins"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportDelete"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExportProcess"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExpiryNotify"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportDelete"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ImportProcess"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Worker started","caller":"jobs/base_workers.go:37","worker":"ExtractContent"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Watcher Started","caller":"jobs/jobs_watcher.go:38"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"info","msg":"Starting schedulers.","caller":"jobs/schedulers.go:47"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_post/scheduler.go:20","scheduler":"last_accessible_post"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"last_accessible_file/scheduler.go:20","scheduler":"last_accessible_file"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Scheduler: isEnabled: true","caller":"notify_admin/install_plugin_scheduler.go:20","scheduler":"install_plugin_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.325 Z","level":"debug","msg":"Cleaning up session store.","caller":"app/server.go:1277"}
{"timestamp":"2023-07-17 23:09:14.507 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:104","filename":"/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2023-07-17 23:09:14.536 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"install_plugin_notify_admin","next_runtime":"2023-07-18 23:09:15.470255335 +0000 UTC m=+86402.530400593"}
{"timestamp":"2023-07-17 23:09:14.538 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:101","error":"resource: License id: "}
{"timestamp":"2023-07-17 23:09:14.539 Z","level":"debug","msg":"Skipping license expiration check because no license is expected on Team Edition","caller":"app/server.go:1379"}
{"timestamp":"2023-07-17 23:09:14.598 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"plugins","next_runtime":"2023-07-18 23:09:15.861378681 +0000 UTC m=+86402.921523936"}
{"timestamp":"2023-07-17 23:09:14.692 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"expiry_notify","next_runtime":"2023-07-17 23:19:15.390785134 +0000 UTC m=+602.450930389"}
{"timestamp":"2023-07-17 23:09:14.772 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"product_notices","next_runtime":"2023-07-18 00:09:14.772598702 +0000 UTC m=+3601.832743959"}
{"timestamp":"2023-07-17 23:09:14.834 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"import_delete","next_runtime":"2023-07-18 23:09:15.729347941 +0000 UTC m=+86402.789493200"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"export_delete","next_runtime":"2023-07-18 23:09:15.164408525 +0000 UTC m=+86402.224553782"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"upgrade_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.896 Z","level":"debug","msg":"Scheduler: isEnabled: false","caller":"notify_admin/scheduler.go:20","scheduler":"trial_notify_admin"}
{"timestamp":"2023-07-17 23:09:14.958 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"2023-07-17 23:10:14.958225723 +0000 UTC m=+62.018370980"}
{"timestamp":"2023-07-17 23:09:15.735 Z","level":"info","msg":"Skipping triggering Elasticsearch channel index fix job as build is not Enterprise ready","caller":"app/migrations.go:570"}
{"timestamp":"2023-07-17 23:09:15.752 Z","level":"debug","msg":"Initializing web routes","caller":"web/web.go:26"}
{"timestamp":"2023-07-17 23:09:15.753 Z","level":"debug","msg":"Rewriting static assets","caller":"utils/subpath.go:83","from_subpath":"/","to_subpath":"/"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Using client directory","caller":"web/static.go:34","clientDir":"/mattermost/client"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:218"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:613","interval_s":"30s"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:299"}
{"timestamp":"2023-07-17 23:09:15.791 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jitsi.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.manland.mattermost-plugin-gitlab.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.github.matterpoll.matterpoll.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/github.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.custom-attributes.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.calls.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/jenkins.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/mattermost-autolink.tar.gz"}
{"timestamp":"2023-07-17 23:09:15.875 Z","level":"info","msg":"Syncing plugin from file store","caller":"app/plugin.go:359","bundle":"plugins/com.mattermost.apps.tar.gz"}
{"timestamp":"2023-07-17 23:09:17.802 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.moussetc.mattermost.plugin.giphy","version":"2.1.0"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64args[plugins/mattermost-autolink/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64pid31"}
{"timestamp":"2023-07-17 23:09:21.220 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"pathplugins/mattermost-autolink/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:21.379 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"address/tmp/plugin2339194495networkunixtimestamp2023-07-17T23:09:21.378Z"}
{"timestamp":"2023-07-17 23:09:21.379 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"mattermost-autolink","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:21.384 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 23:09:21.384 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"mattermost-autolink","version":"1.4.0"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64args[plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64pid43"}
{"timestamp":"2023-07-17 23:09:21.477 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"pathplugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:21.843 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"address/tmp/plugin3892690795networkunixtimestamp2023-07-17T23:09:21.843Z"}
{"timestamp":"2023-07-17 23:09:21.843 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.custom-attributes","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"2023/07/17 23:09:21 LoadPluginConfiguration API failed to unmarshal: json: cannot unmarshal string into Go struct field CustomAttribute.CustomAttributes.GroupIDs of type []string","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.custom-attributes"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 23:09:21.849 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.custom-attributes","version":"1.3.1"}
{"timestamp":"2023-07-17 23:09:25.373 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jenkins","version":"1.1.0"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid51"}
{"timestamp":"2023-07-17 23:09:26.590 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:26.617 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin1700576250networkunixtimestamp2023-07-17T23:09:26.617Z"}
{"timestamp":"2023-07-17 23:09:26.617 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:26.983 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"github","version":"2.1.6"}
{"timestamp":"2023-07-17 23:09:27.058 Z","level":"debug","msg":"ensured bot","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","id":"rx9bx5d3ntrhumudpw3qf387ce","username":"appsbot","plugin_caller":"server/plugin.go:68"}
{"timestamp":"2023-07-17 23:09:27.071 Z","level":"debug","msg":"failed to fetch license twice. May incorrectly default to on-prem mode","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/config/service.go:226"}
{"timestamp":"2023-07-17 23:09:27.071 Z","level":"debug","msg":"configured the plugin.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","commit":"44db542","build_date":"Fri 06 Jan 2023 09:59:54 PM UTC","cloud_mode":"false","developer_mode":"false","allow_http_apps":"true","version":"1.2.0","plugin_caller":"server/plugin.go:92"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"created an AWS client","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","secret":"<nil>","plugin_caller":"upstream/upaws/client.go:121","purpose":"Manifest store","region":"us-east-1","access":"<nil>"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"initialized API and persistent store","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:106"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"HTTP","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 23:09:27.072 Z","level":"debug","msg":"Skipped upstream: AWS Lambda: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"available upstream","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","type":"Mattermost Plugin","plugin_caller":"server/proxy/service.go:232"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"Skipped upstream: OpenFaaS: not configured.","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","error":"OPENFAAS_URL environment variable must be defined: not found","plugin_caller":"server/proxy/service.go:227"}
{"timestamp":"2023-07-17 23:09:27.073 Z","level":"debug","msg":"initialized the app proxy","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:122"}
{"timestamp":"2023-07-17 23:09:27.074 Z","level":"debug","msg":"initialized incoming HTTP","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:125"}
{"timestamp":"2023-07-17 23:09:27.074 Z","level":"info","msg":"activated","caller":"app/plugin_api.go:973","plugin_id":"com.mattermost.apps","plugin_caller":"server/plugin.go:135"}
{"timestamp":"2023-07-17 23:09:27.075 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 23:09:27.075 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.mattermost.apps","version":"1.2.0"}
{"timestamp":"2023-07-17 23:09:27.291 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64args[plugins/jitsi/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:27.317 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64pid64"}
{"timestamp":"2023-07-17 23:09:27.317 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"pathplugins/jitsi/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:27.329 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:27.329 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"jitsi","wrapped_extras":"address/tmp/plugin1683166186networkunixtimestamp2023-07-17T23:09:27.328Z"}
{"timestamp":"2023-07-17 23:09:27.373 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64args[plugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:27.374 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64pid75"}
{"timestamp":"2023-07-17 23:09:27.374 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"pathplugins/com.github.manland.mattermost-plugin-gitlab/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:27.391 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"address/tmp/plugin3158156457networkunixtimestamp2023-07-17T23:09:27.391Z"}
{"timestamp":"2023-07-17 23:09:27.391 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.manland.mattermost-plugin-gitlab","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:27.449 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 23:09:27.449 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"jitsi","version":"2.0.1"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"warn","msg":"ignored duplicate step name","caller":"app/plugin_api.go:979","plugin_id":"com.github.manland.mattermost-plugin-gitlab","name":"oauth-info","flow":"oauth"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 23:09:27.850 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.manland.mattermost-plugin-gitlab","version":"1.6.0"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64args[plugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64pid88"}
{"timestamp":"2023-07-17 23:09:30.530 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:30.868 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:30.868 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"address/tmp/plugin2969576176networkunixtimestamp2023-07-17T23:09:30.867Z"}
{"timestamp":"2023-07-17 23:09:32.362 Z","level":"debug","msg":"Plugin activated","caller":"plugin/environment.go:333","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 23:09:32.362 Z","level":"debug","msg":"Installing plugin","caller":"app/plugin_install.go:410","plugin_id":"com.github.matterpoll.matterpoll","version":"1.5.0"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid101"}
{"timestamp":"2023-07-17 23:09:33.394 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64"}
{"timestamp":"2023-07-17 23:09:33.417 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"address/tmp/plugin543213083networkunixtimestamp2023-07-17T23:09:33.416Z"}
{"timestamp":"2023-07-17 23:09:33.417 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.calls","wrapped_extras":"version1"}
{"timestamp":"2023-07-17 23:09:33.424 Z","level":"debug","msg":"Initializing telemetry","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).initTelemetry telemetry.go:86"}
{"timestamp":"2023-07-17 23:09:33.426 Z","level":"debug","msg":"activating","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).OnActivate activate.go:44"}
{"timestamp":"2023-07-17 23:09:33.432 Z","level":"debug","msg":"cleaning up calls state","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).cleanUpState channel_state.go:251"}
{"timestamp":"2023-07-17 23:09:44.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:10:14.140 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:10:15.023 Z","level":"debug","msg":"Scheduling Job","caller":"migrations/scheduler.go:45","scheduler":"migrations"}
{"timestamp":"2023-07-17 23:10:15.057 Z","level":"debug","msg":"All migrations are complete.","caller":"migrations/scheduler.go:84","scheduler":"migrations"}
{"timestamp":"2023-07-17 23:10:15.089 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"<nil>"}
{"timestamp":"2023-07-17 23:10:44.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}
{"timestamp":"2023-07-17 23:11:14.141 Z","level":"debug","msg":"Email batching job ran. Notifications might be still pending.","caller":"email/email_batching.go:137","number_of_users":0}

If i disable calls plugin - it woks fine.

mustdiechik commented 12 months ago

An interesting behavior: on different clusters, it takes different times for the transition to the Running state. From 40 second to up to 4 minutes. I had similar issue with rabbitMQ release in minikube. It looks like the application can not give the load on the processor. The problem was resolved after I changed cpu governor to "performance" mode. Maybe there is a similar problem here.