search

spantaleev / matrix-docker-ansible-deploy

🐳 Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker
GNU Affero General Public License v3.0
4.88k stars 1.04k forks source link

TASK [matrix-nginx-proxy : Fail if all SSL certificate retrieval attempts failed] #348

Closed dranelixx closed 4 years ago

dranelixx commented 4 years ago

Host System: Ubuntu 18.04

After re running the setup multiple times getting this error.

TASK [matrix-nginx-proxy : Attempt initial SSL certificate retrieval with standalone authenticator (directly)] *******
fatal: [matrix.cddt.net]: FAILED! => {"changed": true, "cmd": "/usr/bin/docker run --rm --name=matrix-certbot --user=991:991 --cap-drop=ALL -p 80:8080 -v /matrix/ssl/config:/etc/letsencrypt -v /matrix/ssl/log:/var/log/letsencrypt certbot/certbot:v1.0.0 certonly --non-interactive --work-dir=/tmp --http-01-port 8080  --standalone --preferred-challenges http --agree-tos --email=adrikonop@gmail.com -d cddt.net", "delta": "0:00:05.663760", "end": "2020-01-19 13:06:29.313933", "msg": "non-zero return code", "rc": 1, "start": "2020-01-19 13:06:23.650173", "stderr": "Saving debug log to /var/log/letsencrypt/letsencrypt.log\nPlugins selected: Authenticator standalone, Installer None\nObtaining a new certificate\nPerforming the following challenges:\nhttp-01 challenge for cddt.net\nWaiting for verification...\nChallenge failed for domain cddt.net\nhttp-01 challenge for cddt.net\nCleaning up challenges\nSome challenges have failed.", "stderr_lines": ["Saving debug log to /var/log/letsencrypt/letsencrypt.log", "Plugins selected: Authenticator standalone, Installer None", "Obtaining a new certificate", "Performing the following challenges:", "http-01 challenge for cddt.net", "Waiting for verification...", "Challenge failed for domain cddt.net", "http-01 challenge for cddt.net", "Cleaning up challenges", "Some challenges have failed."], "stdout": "IMPORTANT NOTES:\n - The following errors were reported by the server:\n\n   Domain: cddt.net\n   Type:   unauthorized\n   Detail: Invalid response from\n   https://cddt.net/.well-known/acme-challenge/ra7e123JYSDq5ey8YaK_7pqCm6VJ6xiNhLL2Sk-_d1o\n   [2a01:238:20a:202:1160::]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD\n   HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not\n   Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p\"\n\n   To fix these errors, please make sure that your domain name was\n   entered correctly and the DNS A/AAAA record(s) for that domain\n   contain(s) the right IP address.", "stdout_lines": ["IMPORTANT NOTES:", " - The following errors were reported by the server:", "", "   Domain: cddt.net", "   Type:   unauthorized", "   Detail: Invalid response from", "   https://cddt.net/.well-known/acme-challenge/ra7e123JYSDq5ey8YaK_7pqCm6VJ6xiNhLL2Sk-_d1o", "   [2a01:238:20a:202:1160::]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD", "   HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not", "   Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p\"", "", "   To fix these errors, please make sure that your domain name was", "   entered correctly and the DNS A/AAAA record(s) for that domain", "   contain(s) the right IP address."]}
...ignoring

fatal: [matrix.cddt.net]: FAILED! => {"changed": true, "cmd": "/usr/bin/docker run --rm --name=matrix-certbot --user=991:991 --cap-drop=ALL -p 127.0.0.1:2402:8080 --network=matrix -v /matrix/ssl/config:/etc/letsencrypt -v /matrix/ssl/log:/var/log/letsencrypt certbot/certbot:v1.0.0 certonly --non-interactive --work-dir=/tmp --http-01-port 8080  --standalone --preferred-challenges http --agree-tos --email=adrikonop@gmail.com -d cddt.net", "delta": "0:00:06.074236", "end": "2020-01-19 13:06:35.773436", "msg": "non-zero return code", "rc": 1, "start": "2020-01-19 13:06:29.699200", "stderr": "Saving debug log to /var/log/letsencrypt/letsencrypt.log\nPlugins selected: Authenticator standalone, Installer None\nObtaining a new certificate\nPerforming the following challenges:\nhttp-01 challenge for cddt.net\nWaiting for verification...\nChallenge failed for domain cddt.net\nhttp-01 challenge for cddt.net\nCleaning up challenges\nSome challenges have failed.", "stderr_lines": ["Saving debug log to /var/log/letsencrypt/letsencrypt.log", "Plugins selected: Authenticator standalone, Installer None", "Obtaining a new certificate", "Performing the following challenges:", "http-01 challenge for cddt.net", "Waiting for verification...", "Challenge failed for domain cddt.net", "http-01 challenge for cddt.net", "Cleaning up challenges", "Some challenges have failed."], "stdout": "IMPORTANT NOTES:\n - The following errors were reported by the server:\n\n   Domain: cddt.net\n   Type:   unauthorized\n   Detail: Invalid response from\n   https://cddt.net/.well-known/acme-challenge/EyeMw4LjNGUneY9-aldelNIbo3ev_orrbI7UYZWGvSU\n   [2a01:238:20a:202:1160::]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD\n   HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not\n   Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p\"\n\n   To fix these errors, please make sure that your domain name was\n   entered correctly and the DNS A/AAAA record(s) for that domain\n   contain(s) the right IP address.", "stdout_lines": ["IMPORTANT NOTES:", " - The following errors were reported by the server:", "", "   Domain: cddt.net", "   Type:   unauthorized", "   Detail: Invalid response from", "   https://cddt.net/.well-known/acme-challenge/EyeMw4LjNGUneY9-aldelNIbo3ev_orrbI7UYZWGvSU", "   [2a01:238:20a:202:1160::]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD", "   HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not", "   Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p\"", "", "   To fix these errors, please make sure that your domain name was", "   entered correctly and the DNS A/AAAA record(s) for that domain", "   contain(s) the right IP address."]}
...ignoring```

TASK [matrix-nginx-proxy : Fail if all SSL certificate retrieval attempts failed] ************************************
fatal: [matrix.cddt.net]: FAILED! => {"changed": false, "msg": "Failed to obtain a certificate directly (by listening on port 80)\nand also failed to obtain by relying on the server at port 80 to proxy the request.\nSee above for details.\nYou may wish to set up proxying of /.well-known/acme-challenge to 2402 or,\nmore easily, stop the server on port 80 while this playbook runs.\n"}

PLAY RECAP ***********************************************************************************************************
matrix.cddt.net            : ok=115  changed=5    unreachable=0    failed=1    skipped=285  rescued=0    ignored=2

Please help

spantaleev commented 4 years ago

You can try making sure there's nothing taking up port 80 on the server.

You can SSH into the server and stop matrix-nginx-proxy manually (systemctl stop matrix-nginx-proxy), before re-running the playbook again.

If you're not using matrix-nginx-proxy (that is, if you have matrix_nginx_proxy_enabled: false in your configuration) and are using your own webserver, try stopping that one too, at least temporarily.

If port 80 is available, then it might be some other problem. Either your domain names are not pointing to the server and Let's Encrypt fails because of that.. Or there's some firewall blocking access to port 80.

dranelixx commented 4 years ago

This is verry wierd and i dont know what is wrong or what im doing

After retrying the Setup it decided to work but the matrix. and the riot. where not reachable so i re run the command an i got again an error fist to check if the server is accessible thro ssh fixed by rerunning but now :

TASK [matrix-base : Ensure APT packages are installed] ***

fatal: [matrix.cddt.net]: FAILED! => {"changed": false, "module_stderr": "FATAL -> Failed to fork.\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 100}

then i re run the setup:

TASK [matrix-base : Ensure APT packages are installed] ***

fatal: [matrix.cddt.net]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"85.214.156.169\". Make sure this host can be reached over ssh: ", "unreachable": true}

one more time re ran the setup:

TASK [matrix-mailer : Ensure mailer environment variables file created] **

fatal: [matrix.cddt.net]: FAILED! => {"msg": "Failed to connect to the host via ssh: "}

and again:

TASK [matrix-base : Ensure APT usage dependencies are installed] *****

fatal: [matrix.cddt.net]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: "}

men i get frustrated

spantaleev commented 4 years ago

Is your server overloaded, running out of disk space, etc.?

dranelixx commented 4 years ago

host or deployment? non of them is overloaded the host system is a fresh install of ubuntu 18.04 with ddclient for pointing domain on server

dranelixx commented 4 years ago

Man i think something is srly broken .... i tryed now 2 trimes to fresh install the host system pointing the domain matrix.cddt.net and riot.cddt.net with ddclient when i ping the domain it shows the right ip of the server then i try ansible-playbook --ask-pass -i inventory/hosts setup.yml --tags=setup-all some times i get an error sometimes not and when i dont get an error and i run ansible-playbook --ask-pass -i inventory/hosts setup.yml --tags=start it runs thro without error: no site is reacheble matrix - riot

dranelixx commented 4 years ago

TASK [Gathering Facts] ** fatal: [matrix.cddt.net]: FAILED! => {"ansible_facts": {}, "changed": false, "msg": "The following modules failed to execute: setup\n setup: [Errno 11] Die Ressource ist zur Zeit nicht verfügbar\n"}

dranelixx commented 4 years ago

The bare domain name which represents your Matrix identity. Matrix user ids for your server will be of the form (@user:<matrix-domain>).

Note: this playbook does not touch the server referenced here. Installation happens on another server ("matrix.").

Example value: example.com matrix_domain: cddt.net

This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.

In case SSL renewal fails at some point, you'll also get an email notification there. If you decide to use another method for managing SSL certifites (different than the default Let's Encrypt), you won't be required to define this variable (see docs/configuring-playbook-ssl-certificates.md).

matrix_ssl_lets_encrypt_support_email: xxxx@gmail.com

A shared secret (between Coturn and Synapse) used for authentication. You can put any string here, but generating a strong one is preferred (e.g. pwgen -s 64 1). matrix_coturn_turn_static_auth_secret: "SECRET"

A secret used to protect access keys issued by the server. You can put any string here, but generating a strong one is preferred (e.g. pwgen -s 64 1). matrix_synapse_macaroon_secret_key: "SECRET"

matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: SECRET

matrix_mautrix_whatsapp_enabled: true

dranelixx commented 4 years ago

[matrix_servers] matrix.cddt.net ansible_host=CORRECTIP ansible_ssh_user=root

dranelixx commented 4 years ago

My final error is:

TASK [Gathering Facts] ** fatal: [matrix.cddt.net]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "cmd": "/sbin/ip -4 route get 8.8.8.8", "deprecations": [{"msg": "Distribution Ubuntu 18.04 on host matrix.cddt.net should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information", "version": "2.12"}], "failed": true, "invocation": {"module_args": {"fact_path": "/etc/ansible/facts.d", "filter": "*", "gather_subset": ["all"], "gather_timeout": 10}}, "msg": "[Errno 11] Die Ressource ist zur Zeit nicht verfügbar", "rc": 11}}, "msg": "The following modules failed to execute: setup\n"}

i did not change anything

dranelixx commented 4 years ago

Now if i try docker ps on the host i get

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f8887ec1e97 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f8887ec1e97
stack: frame={sp:0x7ffd278bff00, fp:0x0} stack=[0x7ffd270c1400,0x7ffd278c0430)
00007ffd278bfe00:  3b30303d63706d2e  67676f2e2a3a3633
00007ffd278bfe10:  2a3a36333b30303d  333b30303d61722e
00007ffd278bfe20:  0000000000000000  0000000000000000
00007ffd278bfe30:  00007f888826f560  00007f8888272bc0
00007ffd278bfe40:  c1162e42fefa39ef  0000000000000000
00007ffd278bfe50:  414fffffe0000000  0000000000000000
00007ffd278bfe60:  2f3a6e69622f6c61  6e6962732f727375
00007ffd278bfe70:  69622f7273752f3a  3a6e6962732f3a6e
00007ffd278bfe80:  73752f3a6e69622f  3a73656d61672f72
00007ffd278bfe90:  636f6c2f7273752f  73656d61672f6c61
00007ffd278bfea0:  2a3a36333b30303d  3b30303d616b6d2e
00007ffd278bfeb0:  33706d2e2a3a3633  2a3a36333b30303d
00007ffd278bfec0:  3b30303d63706d2e  67676f2e2a3a3633
00007ffd278bfed0:  2a3a36333b30303d  333b30303d61722e
00007ffd278bfee0:  3d7661772e2a3a36  2e2a3a36333b3030
00007ffd278bfef0:  333b30303d61676f  7375706f2e2a3a36
00007ffd278bff00: <0000000000000000  3b30303d7870732e
00007ffd278bff10:  7073782e2a3a3633  3a36333b30303d66
00007ffd278bff20:  00007f88888b7740  00007f8884c6f700
00007ffd278bff30:  0000000000001000  00007f8884c6f700
00007ffd278bff40:  00007ffd278c0030  0000000000001000
00007ffd278bff50:  00007ffd278c00c0  00007ffd278bff90
00007ffd278bff60:  000055da0cc5f146 <runtime.step+326>  000055da0f0ed8ab
00007ffd278bff70:  0000000001009357  0000000001009357
00007ffd278bff80:  fffffffe7fffffff  ffffffffffffffff
00007ffd278bff90:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffa0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffb0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffc0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffd0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffe0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bfff0:  ffffffffffffffff  ffffffffffffffff
runtime: unknown pc 0x7f8887ec1e97
stack: frame={sp:0x7ffd278bff00, fp:0x0} stack=[0x7ffd270c1400,0x7ffd278c0430)
00007ffd278bfe00:  3b30303d63706d2e  67676f2e2a3a3633
00007ffd278bfe10:  2a3a36333b30303d  333b30303d61722e
00007ffd278bfe20:  0000000000000000  0000000000000000
00007ffd278bfe30:  00007f888826f560  00007f8888272bc0
00007ffd278bfe40:  c1162e42fefa39ef  0000000000000000
00007ffd278bfe50:  414fffffe0000000  0000000000000000
00007ffd278bfe60:  2f3a6e69622f6c61  6e6962732f727375
00007ffd278bfe70:  69622f7273752f3a  3a6e6962732f3a6e
00007ffd278bfe80:  73752f3a6e69622f  3a73656d61672f72
00007ffd278bfe90:  636f6c2f7273752f  73656d61672f6c61
00007ffd278bfea0:  2a3a36333b30303d  3b30303d616b6d2e
00007ffd278bfeb0:  33706d2e2a3a3633  2a3a36333b30303d
00007ffd278bfec0:  3b30303d63706d2e  67676f2e2a3a3633
00007ffd278bfed0:  2a3a36333b30303d  333b30303d61722e
00007ffd278bfee0:  3d7661772e2a3a36  2e2a3a36333b3030
00007ffd278bfef0:  333b30303d61676f  7375706f2e2a3a36
00007ffd278bff00: <0000000000000000  3b30303d7870732e
00007ffd278bff10:  7073782e2a3a3633  3a36333b30303d66
00007ffd278bff20:  00007f88888b7740  00007f8884c6f700
00007ffd278bff30:  0000000000001000  00007f8884c6f700
00007ffd278bff40:  00007ffd278c0030  0000000000001000
00007ffd278bff50:  00007ffd278c00c0  00007ffd278bff90
00007ffd278bff60:  000055da0cc5f146 <runtime.step+326>  000055da0f0ed8ab
00007ffd278bff70:  0000000001009357  0000000001009357
00007ffd278bff80:  fffffffe7fffffff  ffffffffffffffff
00007ffd278bff90:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffa0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffb0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffc0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffd0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bffe0:  ffffffffffffffff  ffffffffffffffff
00007ffd278bfff0:  ffffffffffffffff  ffffffffffffffff

goroutine 1 [running, locked to thread]:
runtime.asmcgocall(0x55da0e1f03e0, 0xc00006e740)
        /usr/local/go/src/runtime/asm_amd64.s:620 +0x3f fp=0xc00006e728 sp=0xc00006e720 pc=0x55da0cc6e2cf
runtime.newm1(0xc000072e00)
        /usr/local/go/src/runtime/proc.go:1856 +0xa9 fp=0xc00006e768 sp=0xc00006e728 pc=0x55da0cc473e9
runtime.newm(0x55da0ef0c268, 0x0)
        /usr/local/go/src/runtime/proc.go:1840 +0x93 fp=0xc00006e798 sp=0xc00006e768 pc=0x55da0cc47283
runtime.startTemplateThread(...)
        /usr/local/go/src/runtime/proc.go:1876
runtime.main()
        /usr/local/go/src/runtime/proc.go:183 +0x2ba fp=0xc00006e7e0 sp=0xc00006e798 pc=0x55da0cc434aa
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x55da0cc6eb91

rax    0x0
rbx    0x7f888826f840
rcx    0xffffffffffffffff
rdx    0x0
rdi    0x2
rsi    0x7ffd278bff00
rbp    0x55da0e52a608
rsp    0x7ffd278bff00
r8     0x0
r9     0x7ffd278bff00
r10    0x8
r11    0x246
r12    0x55da11a81690
r13    0x11
r14    0x55da0e4cdfb8
r15    0x0
rip    0x7f8887ec1e97
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
dranelixx commented 4 years ago

Sorry for the Spamming but im confused

spantaleev commented 4 years ago

Something seems off with that server, judging by the "can't fork" error above and the "can't create pthread" error you're mentioning now.

Have you tried rebooting the Matrix server?

Also, it sounds like you might be having troubles because of Python 2 vs Python 3 on the server. I'm not sure what to recommend - you can try removing Python 2 and installing Python 3 instead.

spantaleev commented 4 years ago

This may have been caused by #356. Please reopen if you still have this problem!