search

wireapp / wire-server-deploy

Code to install/deploy wire-server (on kubernetes)
https://docs.wire.com
GNU Affero General Public License v3.0
94 stars 45 forks source link

wire-server installation failed #470

Open JonnyX11 opened 3 years ago

JonnyX11 commented 3 years ago

Hello,

I got this error, Anyone know what to do ?

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait

Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition

Can you please help?

arthurwolf commented 3 years ago

If you add --debug to the command, you might get some more information on what's going on.

On Sun, Jun 13, 2021 at 12:20 PM JonnyX11 @.***> wrote:

Hello,

I got this error, Anyone know what to do ?

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait

Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition

Can you please help?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/wireapp/wire-server-deploy/issues/470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2SFN5QQY6TWCH5IG5JXTTSSA6DANCNFSM46TTVQJA .

--

勇気とユーモア

JonnyX11 commented 3 years ago

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug history.go:56: [debug] getting history for release wire-server upgrade.go:123: [debug] preparing upgrade for wire-server upgrade.go:131: [debug] performing update for wire-server upgrade.go:303: [debug] creating upgraded release for wire-server client.go:282: [debug] Starting delete for "cassandra-migrations" Job client.go:122: [debug] creating 1 resource(s) client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0

upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition UPGRADE FAILED main.newUpgradeCmd.func2 helm.sh/helm/v3/cmd/helm/upgrade.go:157 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.1.1/command.go:850 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.1.1/command.go:958 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.1.1/command.go:895 main.main helm.sh/helm/v3/cmd/helm/helm.go:80 runtime.main runtime/proc.go:225 runtime.goexit runtime/asm_amd64.s:1371

JonnyX11 commented 3 years ago

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug history.go:56: [debug] getting history for release wire-server upgrade.go:123: [debug] preparing upgrade for wire-server upgrade.go:131: [debug] performing update for wire-server upgrade.go:303: [debug] creating upgraded release for wire-server client.go:282: [debug] Starting delete for "cassandra-migrations" Job client.go:122: [debug] creating 1 resource(s) client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED client.go:282: [debug] Starting delete for "elasticsearch-index-create" Job client.go:122: [debug] creating 1 resource(s) client.go:491: [debug] Watching for changes to Job elasticsearch-index-create with timeout of 5m0s client.go:519: [debug] Add/Modify event for elasticsearch-index-create: ADDED client.go:558: [debug] elasticsearch-index-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for elasticsearch-index-create: MODIFIED client.go:558: [debug] elasticsearch-index-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition UPGRADE FAILED main.newUpgradeCmd.func2 helm.sh/helm/v3/cmd/helm/upgrade.go:157 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.1.1/command.go:850 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.1.1/command.go:958 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.1.1/command.go:895 main.main helm.sh/helm/v3/cmd/helm/helm.go:80 runtime.main runtime/proc.go:225 runtime.goexit runtime/asm_amd64.s:1371

JonnyX11 commented 3 years ago

Anyone :(

arthurwolf commented 3 years ago

Are you installing the demo stuff, or the production stuff? Which pages in the documentation are you following exactly? Can you give us a complete log of the commands you type, and their output, from the beginning?

I don't have an idea right now of what might be going wrong, but providing this information might help somebody else find a solution for you easier than without that information.

Sorry for taking so long to answer, I completely missed the notification for your message in my inbox.

arthurwolf commented 3 years ago

How much RAM and how many CPUs do you have on the server? Can you give us the results of:

cat /proc/cpuinfo | grep processor | wc -l

free -m

As well as the result of «uptime» before and after the helm upgrade command ?

Thanks.

lucendio commented 3 years ago

Hello @JonnyX11

we are always happy to help, but we kindly ask people who reach out for help to be cooperative. So, would you mind following the template structure that we put in place for these kind of scenarios? You might notice some significant overlap between the questions lisyted in the template and the ones asked by @arthurwolf

Additionally to what is in the template, please provide us with some more context. For example, is it your first installation or are you trying to upgrade an installation? If the latter, did you check the release notes for the new version you installed - also for wire-server? Please note, you will find the code for each chart in there.

Lastly, please provide us with the output of kubectl get pods -o wide --all-namespaces.

Thank you.

JonnyX11 commented 3 years ago

first of all thank you very much for your help!

Basic information

bash-4.4# ansible --version ansible 2.9.12 config file = /wire-server-deploy/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /nix/store/407c5v2x920kri79w0sb9gb1g82q28mg-python3.8-ansible-2.9.12/lib/python3.8/site-packages/ansible executable location = /nix/store/407c5v2x920kri79w0sb9gb1g82q28mg-python3.8-ansible-2.9.12/bin/ansible python version = 3.8.8 (default, Feb 19 2021, 11:04:50) [GCC 10.2.0]

How did you install Wire?

according to the productions docs: https://docs.wire.com/how-to/install/version-requirements.html#persistence https://docs.wire.com/how-to/install/helm-prod.html#helm-prod

I used the docker option for helm https://docs.wire.com/how-to/install/dependencies.html#dependencies

I worked on Ubuntu 18.04 for all virtual machines

Name Amount CPU Cores Memory (GB) Disk Space (GB)
Cassandra 3 2 4 80
MinIO 3 1 2 400
ElasticSearch 3 1 2 60
Kubernetes³ 3 6¹ 8 40
Restund⁴ 2 1 2 10
Per-Server Totals — 11 CPU Cores 18 GB Memory 590 GB Disk Space
Admin Host² 1 1 4 40
Asset Host² 1 1 4 100
Per-Server Totals with Admin and Asset Hosts — 13 CPU Cores 26 GB Memory 730 GB Disk Space


Please see the outputs here :

bash-4.4# kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

bash-4.4# kubectl get pods -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default cassandra-migrations-d5g22 0/1 Completed 0 5m19s 10.233.65.23 kubenode02 default demo-smtp-7f7cc8b585-bcrkq 1/1 Running 4 10d 10.233.64.71 kubenode01 default elasticsearch-index-create-vm6zs 0/1 Init:0/1 3 5m14s 10.233.65.24 kubenode02 default fake-aws-s3-77d9447b8f-2vzgj 1/1 Running 0 66m 10.233.65.19 kubenode02 default fake-aws-s3-reaper-78d9f58dd4-zmqsb 1/1 Running 0 66m 10.233.66.2 kubenode03 default fake-aws-sns-6c7c4b7479-c4thd 2/2 Running 8 10d 10.233.64.74 kubenode01 default fake-aws-sqs-59fbfbcbd4-f7d72 2/2 Running 8 10d 10.233.64.72 kubenode01 default redis-ephemeral-master-0 1/1 Running 4 10d 10.233.64.73 kubenode01 kube-system coredns-7677f9bb54-4kg2d 1/1 Running 16 10d 10.233.65.18 kubenode02 kube-system coredns-7677f9bb54-cnxm2 1/1 Running 4 10d 10.233.64.70 kubenode01 kube-system dns-autoscaler-5b7b5c9b6f-vxwcw 1/1 Running 4 10d 10.233.64.75 kubenode01 kube-system kube-apiserver-kubenode01 1/1 Running 4 10d 172.16.25.80 kubenode01 kube-system kube-apiserver-kubenode02 1/1 Running 18 7d5h 172.16.25.83 kubenode02 kube-system kube-apiserver-kubenode03 1/1 Running 15 7d5h 172.16.25.97 kubenode03 kube-system kube-controller-manager-kubenode01 1/1 Running 0 10d 172.16.25.80 kubenode01 kube-system kube-controller-manager-kubenode02 1/1 Running 0 7d5h 172.16.25.83 kubenode02 kube-system kube-controller-manager-kubenode03 1/1 Running 0 7d5h 172.16.25.97 kubenode03 kube-system kube-flannel-dwpxs 1/1 Running 4 10d 172.16.25.80 kubenode01 kube-system kube-flannel-jdtgg 1/1 Running 16 7d5h 172.16.25.83 kubenode02 kube-system kube-flannel-lv8kw 1/1 Running 13 7d5h 172.16.25.97 kubenode03 kube-system kube-proxy-4k4qz 1/1 Running 0 3h34m 172.16.25.80 kubenode01 kube-system kube-proxy-4njtd 1/1 Running 0 3h34m 172.16.25.83 kubenode02 kube-system kube-proxy-dlhcc 1/1 Running 0 3h34m 172.16.25.97 kubenode03 kube-system kube-scheduler-kubenode01 1/1 Running 0 10d 172.16.25.80 kubenode01 kube-system kube-scheduler-kubenode02 1/1 Running 0 7d5h 172.16.25.83 kubenode02 kube-system kube-scheduler-kubenode03 1/1 Running 0 7d5h 172.16.25.97 kubenode03 kube-system nodelocaldns-gfvb7 1/1 Running 15 7d5h 172.16.25.97 kubenode03 kube-system nodelocaldns-jr6ft 1/1 Running 16 7d5h 172.16.25.83 kubenode02 kube-system nodelocaldns-vwhzg 1/1 Running 4 10d 172.16.25.80 kubenode01

bash-4.4# cat /proc/cpuinfo | grep processor | wc -l 4 bash-4.4# bash-4.4# free -m total used free shared buff/cache available Mem: 16013 579 12578 1 2855 15135 Swap: 2047 0 2047 bash-4.4#

bash-4.4# helm upgrade --install wire-server wire/wire-server -f my-wire-server/values.yaml -f my-wire-server/secrets.yaml --wait --debug history.go:56: [debug] getting history for release wire-server upgrade.go:123: [debug] preparing upgrade for wire-server upgrade.go:131: [debug] performing update for wire-server upgrade.go:303: [debug] creating upgraded release for wire-server client.go:282: [debug] Starting delete for "cassandra-migrations" Job client.go:122: [debug] creating 1 resource(s) client.go:491: [debug] Watching for changes to Job cassandra-migrations with timeout of 5m0s client.go:519: [debug] Add/Modify event for cassandra-migrations: ADDED client.go:558: [debug] cassandra-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED client.go:558: [debug] cassandra-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for cassandra-migrations: MODIFIED client.go:282: [debug] Starting delete for "elasticsearch-index-create" Job client.go:122: [debug] creating 1 resource(s) client.go:491: [debug] Watching for changes to Job elasticsearch-index-create with timeout of 5m0s client.go:519: [debug] Add/Modify event for elasticsearch-index-create: ADDED client.go:558: [debug] elasticsearch-index-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go:519: [debug] Add/Modify event for elasticsearch-index-create: MODIFIED client.go:558: [debug] elasticsearch-index-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 upgrade.go:369: [debug] warning: Upgrade "wire-server" failed: pre-upgrade hooks failed: timed out waiting for the condition Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition helm.go:81: [debug] pre-upgrade hooks failed: timed out waiting for the condition UPGRADE FAILED main.newUpgradeCmd.func2 helm.sh/helm/v3/cmd/helm/upgrade.go:157 github.com/spf13/cobra.(Command).execute github.com/spf13/cobra@v1.1.1/command.go:850 github.com/spf13/cobra.(Command).ExecuteC github.com/spf13/cobra@v1.1.1/command.go:958 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.1.1/command.go:895 main.main helm.sh/helm/v3/cmd/helm/helm.go:80 runtime.main runtime/proc.go:225 runtime.goexit runtime/asm_amd64.s:1371

bash-4.4# helm list --all NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION cassandra-external default 2 2021-06-20 11:43:21.448671888 +0000 UTC deployed cassandra-external-2.108.0
databases-ephemeral default 2 2021-06-20 11:43:41.528915369 +0000 UTC deployed databases-ephemeral-2.108.0
elasticsearch-external default 2 2021-06-20 11:43:33.670740997 +0000 UTC deployed elasticsearch-external-2.108.0
fake-aws default 2 2021-06-20 11:49:41.841649782 +0000 UTC deployed fake-aws-2.108.0
minio-external default 4 2021-06-20 11:45:08.514165984 +0000 UTC deployed minio-external-2.108.0
smtp default 2 2021-06-20 11:54:14.683598062 +0000 UTC deployed demo-smtp-2.108.0 1.0
wire-server default 9 2021-06-20 12:50:43.972757468 +0000 UTC failed wire-server-2.108.0

JonnyX11 commented 3 years ago

Hello @JonnyX11

we are always happy to help, but we kindly ask people who reach out for help to be cooperative. So, would you mind following the template structure that we put in place for these kind of scenarios? You might notice some significant overlap between the questions lisyted in the template and the ones asked by @arthurwolf

Additionally to what is in the template, please provide us with some more context. For example, is it your first installation or are you trying to upgrade an installation? If the latter, did you check the release notes for the new version you installed - also for wire-server? Please note, you will find the code for each chart in there.

Lastly, please provide us with the output of kubectl get pods -o wide --all-namespaces.

Thank you.

The details was OK on the last post ? it was clear ?

JonnyX11 commented 3 years ago

Anyone?