search

getodk / central

ODK Central is a server that is easy to use, very fast, and stuffed with features that make data collection easier. Contribute and make the world a better place! ✨🗄✨
https://docs.getodk.org/central-intro/
Apache License 2.0
125 stars 151 forks source link

Unable to Run After Updating to 0.4 #56

Closed arkhub closed 5 years ago

arkhub commented 5 years ago

After updating odk central to 0.4 and following the readme, I managed to run odk-central using docker-compose up -d, but somehow odk-central cannot be accessed in browsers with "This site can’t be reached" error. Is there any step in updating that I missed?

edit : i try to start docker-compose and here's the snippet of the log I suspect the culprit : 

nginx_1     | 2019/02/17 11:03:44 [warn] 20#20: duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
nginx_1     | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
nginx_1     | 2019/02/17 11:03:44 [emerg] 20#20: a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
nginx_1     | nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
issa-tseng commented 5 years ago

ugh, hm, how did that happen? i'll take a look tomorrow.

arkhub commented 5 years ago

Actually I didn't really catch the first few errors until I found out my domain ssl registration check in letsencrypt has been blocked due too too many request (probably nginx trying multiple times after unsuccessfull restart). So I switched to customssl from cloudflare and change the .env settings (just to clarify i had to set the domain to "local" is it right? and ssl to "customssl").

Then the above errors occured even though I already remove the images and container beforehand and start docker-compose build again.

update : I fix the first 2 line warning above by removing "text/html" in odk.conf.template (turns out gzip already enable text/html by default so didn't need to be mentioned) but I still I haven't find out where the settings in "/etc/nginx/sites-enabled/default".

issa-tseng commented 5 years ago

thanks for the update—mostly i am still confused why we didn't see this when we did our own upgrade and fresh install testing.

yanokwa commented 5 years ago

Looks like a similar issue has been reported at https://forum.opendatakit.org/t/odk-central-v0-4-setup-duplicate-default-server-error/17942.

issa-tseng commented 5 years ago

still looking into this. for whatever reason none of our own installations are showing this behavior, but we have 2 or possibly 3 reports of what you're seeing. would you mind running the diagnostics i suggest here and seeing what you get?

arkhub commented 5 years ago

sure, is there any command i have to run?

btw, I've tried deleting all previous docker with docker prune system (-a and --volumes) and restart the installation process from start using root user. Here's the log in systemctl status docker-compose@central before the nginx exited with process 1

Feb 19 03:11:33 <my domain> docker-compose[7897]: service_1   | starting server.
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | .......................................................................................................................................................................+.
Feb 19 03:12:20 <my domain> docker-compose[7897]: ......................................+.....................................+.....+...........................................................+........................................
Feb 19 03:12:20 <my domain> docker-compose[7897]: .......................................................................................................+.........................+.....................................................
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | writing a new nginx configuration file..
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | 2019/02/18 20:12:20 [warn] 19#19: duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | 2019/02/18 20:12:20 [emerg] 19#19: a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
Feb 19 03:12:20 <my domain> docker-compose[7897]: nginx_1     | nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22

and here's the log when I run using letsencrypt in .env

nginx_1     | writing a new nginx configuration file..
nginx_1     | Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf
nginx_1     | Keyfile(s) missing for /etc/nginx/conf.d/odk.conf, disabling...
nginx_1     | Done with startup
nginx_1     | Run certbot
nginx_1     | + parse_domains
nginx_1     | + xargs echo
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\/etc/letsencrypt/live/\(.*\)/privkey.pem;&\1&p /etc/nginx/conf.d/certbot.conf
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\/etc/letsencrypt/live/\(.*\)/privkey.pem;&\1&p /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + xargs echo
nginx_1     | + is_renewal_required <my domain name>
nginx_1     | + last_renewal_file=/etc/letsencrypt/live/<my domain name>/privkey.pem
nginx_1     | + [ ! -e /etc/letsencrypt/live/<my domain name>/privkey.pem ]
nginx_1     | + return
nginx_1     | + get_certificate <my domain name> <my email address>
nginx_1     | + echo Getting certificate for domain <my domain name> on behalf of user <my email address>
nginx_1     | Getting certificate for domain <my domain name> on behalf of user <my email address>
nginx_1     | Production ...
nginx_1     | running certbot ... https://acme-v01.api.letsencrypt.org/directory <my domain name> <my email address>
nginx_1     | + PRODUCTION_URL=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + STAGING_URL=https://acme-staging.api.letsencrypt.org/directory
nginx_1     | + [  = 1 ]
nginx_1     | + letsencrypt_url=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + echo Production ...
nginx_1     | + echo running certbot ... https://acme-v01.api.letsencrypt.org/directory <my domain name> <my email address>
nginx_1     | + certbot certonly --agree-tos --keep -n --text --email <my email address> --server https://acme-v01.api.letsencrypt.org/directory -d <my domain name> --http-01-port 1337 --standalone --preferred-challenges http-01 --debug
nginx_1     | 2019/02/18 20:32:18 [emerg] 92#92: a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
nginx_1     | nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
nginx_1     | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx_1     | Plugins selected: Authenticator standalone, Installer None
nginx_1     | Obtaining a new certificate
nginx_1     | Performing the following challenges:
nginx_1     | http-01 challenge for <my domain name>
nginx_1     | Waiting for verification...
nginx_1     | Cleaning up challenges
nginx_1     | Exiting abnormally:
nginx_1     | Traceback (most recent call last):
nginx_1     |   File "/usr/local/bin/certbot", line 11, in <module>
nginx_1     |     sys.exit(main())
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1364, in main
nginx_1     |     return config.func(config, plugins)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1254, in certonly
nginx_1     |     lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 120, in _get_and_save_cert
nginx_1     |     lineage = le_client.obtain_and_enroll_certificate(domains, certname)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 391, in obtain_and_enroll_certificate
nginx_1     |     cert, chain, key, _ = self.obtain_certificate(domains)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 334, in obtain_certificate
nginx_1     |     orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 370, in _get_order_and_authorizations
nginx_1     |     authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
nginx_1     |     self._respond(aauthzrs, resp, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 155, in _respond
nginx_1     |     self._poll_challenges(aauthzrs, chall_update, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 226, in _poll_challenges
nginx_1     |     raise errors.FailedChallenges(all_failed_achalls)
nginx_1     | FailedChallenges: Failed authorization procedure. <my domain name> (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://<my domain name>/.well-known/acme-challenge/<token code>: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
nginx_1     | Please see the logfiles in /var/log/letsencrypt for more details.
nginx_1     | IMPORTANT NOTES:
nginx_1     |  - The following errors were reported by the server:
nginx_1     |
nginx_1     |    Domain: <my domain name>
nginx_1     |    Type:   unauthorized
nginx_1     |    Detail: Invalid response from
nginx_1     |    http://<my domain name>/.well-known/acme-challenge/<token code>:
nginx_1     |    "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\"
nginx_1     |    lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js
nginx_1     |    "
nginx_1     |
nginx_1     |    To fix these errors, please make sure that your domain name was
nginx_1     |    entered correctly and the DNS A/AAAA record(s) for that domain
nginx_1     |    contain(s) the right IP address.
nginx_1     |  - Your account credentials have been saved in your Certbot
nginx_1     |    configuration directory at /etc/letsencrypt. You should make a
nginx_1     |    secure backup of this folder now. This configuration directory will
nginx_1     |    also contain certificates and private keys obtained by Certbot so
nginx_1     |    making regular backups of this folder is ideal.
nginx_1     | + error Cerbot failed for <my domain name>. Check the logs for details.
nginx_1     | + set +x
nginx_1     | Cerbot failed for <my domain name>. Check the logs for details.
nginx_1     | + exit_code=1
nginx_1     | + auto_enable_configs
nginx_1     | + keyfiles_exist /etc/nginx/conf.d/certbot.conf
nginx_1     | + parse_keyfiles /etc/nginx/conf.d/certbot.conf
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\(.*\);&\1&p /etc/nginx/conf.d/certbot.conf
nginx_1     | + return 0
nginx_1     | + [ conf = nokey ]
nginx_1     | + keyfiles_exist /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + parse_keyfiles /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\(.*\);&\1&p /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + [ ! -f /etc/letsencrypt/live/<my domain name>/privkey.pem ]
nginx_1     | + echo Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + return 1
nginx_1     | Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + [ nokey = conf ]
nginx_1     | + kill -HUP 92
nginx_1     | /scripts/run_certbot.sh: 33: kill: No such process
nginx_1     |
nginx_1     | + set +x

looks like on both occasion the nginx failed to start, on the letsencrypt case it failed certbot authentication because invalid response from the webserver.

here's the debian server version i use : 

Distributor ID: Debian
Description:    Debian GNU/Linux 8.11 (jessie)
Release:        8.11
Codename:       jessie

while I'm at it, is it compatible to use newer debian version (stretch - 9)? while the odk-central is in docker environment I just want to make sure it stays compatible.

issa-tseng commented 5 years ago

okay, i think i've found the issue (#57); would you mind pulling the latest central from git and trying again? you may have to rm the nginx container again.

arkhub commented 5 years ago

trying from fresh install in debian 9 as root, this time the error is different

Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + keyfiles_exist /etc/nginx/conf.d/odk.conf.nokey
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + parse_keyfiles /etc/nginx/conf.d/odk.conf.nokey
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\(.*\);&\1&p /etc/nginx/conf.d/odk.conf.nokey
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | Couldn't find keyfile /etc/letsencrypt/live/www.<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + [ ! -f /etc/letsencrypt/live/www.<my domain name>/privkey.pem ]
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + echo Couldn't find keyfile /etc/letsencrypt/live/www.<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + return 1
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + [ nokey = conf ]
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + kill -HUP 20
Feb 19 12:56:55 <my domain name> docker-compose[6892]: nginx_1     | + set +x

I set the .env using letsencrypt again, I'll ty using customssl

edit : using customssl, now server finally up again, only the warning still persist

Feb 19 13:31:12 <my domain name> docker-compose[14086]: nginx_1     | 2019/02/19 13:31:12 [warn] 9#9: duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
Feb 19 13:31:12 <my domain name> docker-compose[14086]: nginx_1     | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
Feb 19 13:31:12 <my domain name> docker-compose[14086]: nginx_1     | 2019/02/19 13:31:12 [warn] 9#9: could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; igno
Feb 19 13:31:12 <my domain name> docker-compose[14086]: nginx_1     | nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_has
issa-tseng commented 5 years ago

the failure there would have happened earlier in the log, when letsencrypt tried to run. if you’re still interested in a resolution here, would you mind finding that and pasting it? On Tue, Feb 19, 2019 at 05:08 Ark notifications@github.com wrote:

Reopened #56 https://github.com/opendatakit/central/issues/56.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/opendatakit/central/issues/56#event-2148798522, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHMAppCqwdwTQxKkvGZiFvVIIm2OLY5ks5vO_dagaJpZM4a_e1g .

arkhub commented 5 years ago

the failure there would have happened earlier in the log, when letsencrypt tried to run. if you’re still interested in a resolution here, would you mind finding that and pasting it? … On Tue, Feb 19, 2019 at 05:08 Ark @.***> wrote: Reopened #56 <#56>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHMAppCqwdwTQxKkvGZiFvVIIm2OLY5ks5vO_dagaJpZM4a_e1g .

arkhub commented 5 years ago

the failure there would have happened earlier in the log, when letsencrypt tried to run. if you’re still interested in a resolution here, would you mind finding that and pasting it?

sure, do you want the error log from last docker-compose run when using letsencrypt or is there something else I need to paste here?

edit : i need to replace my mouse from kept closing and opening this report

arkhub commented 5 years ago

for this warning :

nginx_1     | 2019/02/19 13:31:12 [warn] 9#9: duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
nginx_1     | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22

may I suggest removing the "text/html" in : central/files/nginx/odk.cont.template line 22 like this : gzip_types text/plain text/css application/json application/x-javascript text/xml text/csv; (or if you want I can try pull request).

as for this logs : 

nginx_1     | 2019/02/19 13:31:12 [warn] 9#9: could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size
nginx_1     | nginx: [warn] could not build optimal variables_hash, you should increase either variables_hash_max_size: 1024 or variables_hash_bucket_size: 64; ignoring variables_hash_bucket_size

might it be related to why letsencrypt failed?

edit : I also have this warning when I use docker-compose ps (run as customssl from cloudflare with domain set as "local")

WARNING: The DOMAIN variable is not set. Defaulting to a blank string.
WARNING: The SSL_TYPE variable is not set. Defaulting to a blank string.
WARNING: The SYSADMIN_EMAIL variable is not set. Defaulting to a blank string.
issa-tseng commented 5 years ago

as for the former, i agree; we will do that for the next release. but it probably doesn't hurt anything for it to be present for the time being.

as for the optimal variables_hash warning being related to letsencrypt, mayybe. but if letsencrypt failed, you should see that failure earlier in the log somewhere. as an example, here is the successful run on my server:

nginx_1     | Production ...
nginx_1     | running certbot ... https://acme-v01.api.letsencrypt.org/directory odk.antinod.es clint@dontexplain.com
nginx_1     | + is_renewal_required odk.antinod.es
nginx_1     | + last_renewal_file=/etc/letsencrypt/live/odk.antinod.es/privkey.pem
nginx_1     | + [ ! -e /etc/letsencrypt/live/odk.antinod.es/privkey.pem ]
nginx_1     | + return
nginx_1     | + get_certificate odk.antinod.es clint@dontexplain.com
nginx_1     | + echo Getting certificate for domain odk.antinod.es on behalf of user clint@dontexplain.com
nginx_1     | + PRODUCTION_URL=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + STAGING_URL=https://acme-staging.api.letsencrypt.org/directory
nginx_1     | + [  = 1 ]
nginx_1     | + letsencrypt_url=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + echo Production ...
nginx_1     | + echo running certbot ... https://acme-v01.api.letsencrypt.org/directory odk.antinod.es clint@dontexplain.com
nginx_1     | + certbot certonly --agree-tos --keep -n --text --email clint@dontexplain.com --server https://acme-v01.api.letsencrypt.org/directory -d odk.antinod.es --http-01-port 1337 --standalone --preferred-challenges http-01 --debug
nginx_1     | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx_1     | Plugins selected: Authenticator standalone, Installer None
nginx_1     | Obtaining a new certificate
nginx_1     | Performing the following challenges:
nginx_1     | http-01 challenge for odk.antinod.es
nginx_1     | Waiting for verification...
nginx_1     | 66.133.109.36 - - [18/Feb/2019:22:53:09 +0000] "GET /.well-known/acme-challenge/{redacted} HTTP/1.1" 200 98 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

when it was failing, instead of the GET line at the end from the actual nginx process itself, i got a failure message.

as for the docker-compose ps warning, that message appears when you run docker-compose commands from anywhere other than the central/ project repository.

arkhub commented 5 years ago

Few question before I'll try to recreate the previous error when using letsecnrypt :

  1. Do I need to delete "privkey.pem" and "fullchain.pem" in customssl folder if I want to switch to letsencrypt?
  2. Is it necessary to delete the containers and volumes or just run "docker-compose build"?
issa-tseng commented 5 years ago

it is not necessary to delete those files. it should not be necessary to delete containers and volumes; just running build should do it.

arkhub commented 5 years ago

Sorry for late reply, here's the error log when running using letsencrypt : 

nginx_1     | writing a new nginx configuration file..
nginx_1     | Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf
nginx_1     | Keyfile(s) missing for /etc/nginx/conf.d/odk.conf, disabling...
nginx_1     | Done with startup
nginx_1     | Run certbot
nginx_1     | + parse_domains
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\/etc/letsencrypt/live/\(.*\)/privkey.pem;&\1&p /etc/nginx/conf.d/certbot.conf
nginx_1     | + xargs echo
nginx_1     | + xargs echo
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\/etc/letsencrypt/live/\(.*\)/privkey.pem;&\1&p /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + is_renewal_required <my domain name>
nginx_1     | + last_renewal_file=/etc/letsencrypt/live/<my domain name>/privkey.pem
nginx_1     | + [ ! -e /etc/letsencrypt/live/<my domain name>/privkey.pem ]
nginx_1     | + return
nginx_1     | + get_certificate <my domain name> <my email>
nginx_1     | + echo Getting certificate for domain <my domain name> on behalf of user <my email>
nginx_1     | Getting certificate for domain <my domain name> on behalf of user <my email>
nginx_1     | + PRODUCTION_URL=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + STAGING_URL=https://acme-staging.api.letsencrypt.org/directory
nginx_1     | Production ...
nginx_1     | running certbot ... https://acme-v01.api.letsencrypt.org/directory <my domain name> <my email>
nginx_1     | + [  = 1 ]
nginx_1     | + letsencrypt_url=https://acme-v01.api.letsencrypt.org/directory
nginx_1     | + echo Production ...
nginx_1     | + echo running certbot ... https://acme-v01.api.letsencrypt.org/directory <my domain name> <my email>
nginx_1     | + certbot certonly --agree-tos --keep -n --text --email <my email> --server https://acme-v01.api.letsencrypt.org/directory -d <my domain name> --http-01-port 1337 --standalone --preferred-challenges http-01 --debug
nginx_1     | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx_1     | Plugins selected: Authenticator standalone, Installer None
nginx_1     | Obtaining a new certificate
nginx_1     | Performing the following challenges:
nginx_1     | http-01 challenge for <my domain name>
nginx_1     | Waiting for verification...
nginx_1     | Cleaning up challenges
nginx_1     | Exiting abnormally:
nginx_1     | Traceback (most recent call last):
nginx_1     |   File "/usr/local/bin/certbot", line 11, in <module>
nginx_1     |     sys.exit(main())
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1364, in main
nginx_1     |     return config.func(config, plugins)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1254, in certonly
nginx_1     |     lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 120, in _get_and_save_cert
nginx_1     |     lineage = le_client.obtain_and_enroll_certificate(domains, certname)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 391, in obtain_and_enroll_certificate
nginx_1     |     cert, chain, key, _ = self.obtain_certificate(domains)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 334, in obtain_certificate
nginx_1     |     orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 370, in _get_order_and_authorizations
nginx_1     |     authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
nginx_1     |     self._respond(aauthzrs, resp, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 155, in _respond
nginx_1     |     self._poll_challenges(aauthzrs, chall_update, best_effort)
nginx_1     |   File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 226, in _poll_challenges
nginx_1     |     raise errors.FailedChallenges(all_failed_achalls)
nginx_1     | FailedChallenges: Failed authorization procedure. <my domain name> (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://<my domain name>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
nginx_1     | Please see the logfiles in /var/log/letsencrypt for more details.
nginx_1     | IMPORTANT NOTES:
nginx_1     |  - The following errors were reported by the server:
nginx_1     |
nginx_1     |    Domain: <my domain name>
nginx_1     |    Type:   unauthorized
nginx_1     |    Detail: Invalid response from
nginx_1     |    http://<my domain name>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs:
nginx_1     |    "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\"
nginx_1     |    lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js
nginx_1     |    "
nginx_1     |
nginx_1     |    To fix these errors, please make sure that your domain name was
nginx_1     |    entered correctly and the DNS A/AAAA record(s) for that domain
nginx_1     |    contain(s) the right IP address.
nginx_1     |  - Your account credentials have been saved in your Certbot
nginx_1     |    configuration directory at /etc/letsencrypt. You should make a
nginx_1     |    secure backup of this folder now. This configuration directory will
nginx_1     |    also contain certificates and private keys obtained by Certbot so
nginx_1     |    making regular backups of this folder is ideal.
nginx_1     | + error Cerbot failed for <my domain name>. Check the logs for details.
nginx_1     | + set +x
nginx_1     | Cerbot failed for <my domain name>. Check the logs for details.
nginx_1     | + exit_code=1
nginx_1     | + auto_enable_configs
nginx_1     | + keyfiles_exist /etc/nginx/conf.d/certbot.conf
nginx_1     | + parse_keyfiles /etc/nginx/conf.d/certbot.conf
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\(.*\);&\1&p /etc/nginx/conf.d/certbot.conf
nginx_1     | + return 0
nginx_1     | + [ conf = nokey ]
nginx_1     | + keyfiles_exist /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + parse_keyfiles /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + sed -n -e s&^\s*ssl_certificate_key\s*\(.*\);&\1&p /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + [ ! -f /etc/letsencrypt/live/<my domain name>/privkey.pem ]
nginx_1     | + echo Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + return 1
nginx_1     | + [ nokey = conf ]
nginx_1     | Couldn't find keyfile /etc/letsencrypt/live/<my domain name>/privkey.pem for /etc/nginx/conf.d/odk.conf.nokey
nginx_1     | + kill -HUP 17
nginx_1     | + set +x

there's no change in my domain name and previously it just works with letsencrypt, the error somewhat related to failed challange (response) from the nginx server?

issa-tseng commented 5 years ago

so, the problem here is that the domain name needs to be set in the .env file in your central/ directory; it appears to be set to the value <my domain name>, resulting in this:

Invalid response from http://<my domain name>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs
arkhub commented 5 years ago

so, the problem here is that the domain name needs to be set in the .env file in your central/ directory; it appears to be set to the value <my domain name>, resulting in this:

Invalid response from http://<my domain name>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs

sorry forgot to tell that and is indeed set with my domain name and email in the .env files, I just find and replace it before posting here.

issa-tseng commented 5 years ago

oh, okay, i see. hmm, what happens if you run

docker-compose exec nginx cat /var/log/letsencrypt/letsencrypt.log from inside the central/ project dir?

arkhub commented 5 years ago 
2019-02-20 18:55:42,338:DEBUG:certbot.main:certbot version: 0.27.1
2019-02-20 18:55:42,339:DEBUG:certbot.main:Arguments: ['--agree-tos', '--keep', '-n', '--text', '--email', '<my email>', '--server', 'https://acme-v01.api.letsencrypt.org/directory', '-d', '<my domain>', '--http-01-port', '1337', '--standalone', '--preferred-challenges', 'http-01', '--debug']
2019-02-20 18:55:42,339:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-02-20 18:55:42,363:DEBUG:certbot.log:Root logging level set at 20
2019-02-20 18:55:42,364:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-02-20 18:55:42,366:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2019-02-20 18:55:42,443:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f20d0106b10>
Prep: True
2019-02-20 18:55:42,446:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7f20d0106b10> and installer None
2019-02-20 18:55:42,446:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2019-02-20 18:55:42,681:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2019-02-20 18:55:42,715:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org:443
2019-02-20 18:55:43,101:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2019-02-20 18:55:43,103:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
Replay-Nonce: gqJJJNftoiFGhCt_vp4RRZ4IUm3brIicrJ6IvR63tJQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 20 Feb 2019 18:55:43 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:43 GMT
Connection: keep-alive

{
  "i9boTo1bBbg": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}
2019-02-20 18:55:43,104:DEBUG:acme.client:Requesting fresh nonce
2019-02-20 18:55:43,104:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-reg.
2019-02-20 18:55:43,355:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "HEAD /acme/new-reg HTTP/1.1" 405 0
2019-02-20 18:55:43,356:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Replay-Nonce: r3bDNyWBz6vT09fgAxMeaAdPOdrSodDJxQJWo0Epc2w
Expires: Wed, 20 Feb 2019 18:55:43 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:43 GMT
Connection: keep-alive

2019-02-20 18:55:43,356:DEBUG:acme.client:Storing nonce: r3bDNyWBz6vT09fgAxMeaAdPOdrSodDJxQJWo0Epc2w
2019-02-20 18:55:43,356:DEBUG:acme.client:JWS payload:
{
  "contact": [
    "mailto:<my email>"
  ],
  "resource": "new-reg"
}
2019-02-20 18:55:43,361:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-reg:
{
  "protected": "eyJub25jZSI6ICJyM2JETnlXQno2dlQwOWZnQXhNZWFBZFBPZHJTb2RESnhRSldvMEVwYzJ3IiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAic0RtV2x1VXlhbWtHWlRmazd2QXFuQkxERzI5VzA0RHBuTjBYVWV4M3VZeU9rTnZGYlBBSVFZdzdQbHlaN3BXQTRkbU1Fd2RvblNfNUVOcHVOZ2VLV05NV2tMaWgxaEl5Z0s3czlmak1IOEFPeV9ma1RheV9BeV84RWlSTW4xakJjZlRFYXh5c3ZwdDZOdWFuTE12Mmt6M3cxaExTQWtOU3dRZ245dWZndHBNM01mSm1kQlZQRE5wWGFxRDJuQ0VBRVExSndqSWdZV0R5T2lBNmlGTC1QQVZTTS1WaWJYMFV0cjhHQ3dBNHZQWGhSNHY1OGlmOGJaWFlMeUhrcWNhTDhtUGowRU9mdkNvUldtTFFLa1lZSnhFMHFuZFQzenJickZYS1Z4dkNwZGhiUUh6eTBKd3lGUEM0a1Q0QkxRRkZpRUVJTHp0bDhtU3VwWGV3emFDS3BRIn19",
  "payload": "ewogICJjb250YWN0IjogWwogICAgIm1haWx0bzphcmsucGhvbmVAZ21haWwuY29tIgogIF0sIAogICJyZXNvdXJjZSI6ICJuZXctcmVnIgp9",
  "signature": "AgwMsie2AFBNWkeBxs8dKeL_OZoQ26YJ8TDNLdCYxf5WJdl_p-21OkrJWbMN7r7g0JdUQ8fJzs-sfCou4-I9DznDvXvJQDv-ZlNAIE14IktayjPW02kfkXx2qUUEaENJs9i1SR4OAW2dkozhAmavoIRqQLwdUIGO8CKE_T91R2Oj1izMnTGlP7T4qu0aUTpuLRnCget-8WToFn345McDSkTJ3zaunJ5q0m6XSRAA2J8jA1WLfTrWM8kl_GRO3Ao1Wdn-Uae32IPMWfl1sR2W0vhDp1DMsmxrscmDf7P0y4FoP3eVZgfG-TnWSdQ18icLRdBBvAc2YRjyhfLtoCaRrg"
}
2019-02-20 18:55:43,628:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/new-reg HTTP/1.1" 201 580
2019-02-20 18:55:43,629:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 580
Boulder-Requester: 51894479
Link: <https://acme-v01.api.letsencrypt.org/acme/new-authz>;rel="next", <https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel="terms-of-service"
Location: https://acme-v01.api.letsencrypt.org/acme/reg/51894479
Replay-Nonce: JDsKOa4VWNIDGqa44M98gf1q2DlKBO0yC2JBdG0IIeU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 20 Feb 2019 18:55:43 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:43 GMT
Connection: keep-alive

{
  "id": 51894479,
  "key": {
    "kty": "RSA",
    "n": "sDmWluUyamkGZTfk7vAqnBLDG29W04DpnN0XUex3uYyOkNvFbPAIQYw7PlyZ7pWA4dmMEwdonS_5ENpuNgeKWNMWkLih1hIygK7s9fjMH8AOy_fkTay_Ay_8EiRMn1jBcfTEaxysvpt6NuanLMv2kz3w1hLSAkNSwQgn9ufgtpM3MfJmdBVPDNpXaqD2nCEAEQ1JwjIgYWDyOiA6iFL-PAVSM-VibX0Utr8GCwA4vPXhR4v58if8bZXYLyHkqcaL8mPj0EOfvCoRWmLQKkYYJxE0qndT3zrbrFXKVxvCpdhbQHzy0JwyFPC4kT4BLQFFiEEILztl8mSupXewzaCKpQ",
    "e": "AQAB"
  },
  "contact": [
    "mailto:<my email>"
  ],
  "initialIp": "194.59.165.91",
  "createdAt": "2019-02-20T18:55:43.489204255Z",
  "status": "valid"
}
2019-02-20 18:55:43,629:DEBUG:acme.client:Storing nonce: JDsKOa4VWNIDGqa44M98gf1q2DlKBO0yC2JBdG0IIeU
2019-02-20 18:55:43,631:DEBUG:acme.client:JWS payload:
{
  "status": "valid",
  "contact": [
    "mailto:<my email>"
  ],
  "resource": "reg",
  "agreement": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
  "key": {
    "e": "AQAB",
    "kty": "RSA",
    "n": "sDmWluUyamkGZTfk7vAqnBLDG29W04DpnN0XUex3uYyOkNvFbPAIQYw7PlyZ7pWA4dmMEwdonS_5ENpuNgeKWNMWkLih1hIygK7s9fjMH8AOy_fkTay_Ay_8EiRMn1jBcfTEaxysvpt6NuanLMv2kz3w1hLSAkNSwQgn9ufgtpM3MfJmdBVPDNpXaqD2nCEAEQ1JwjIgYWDyOiA6iFL-PAVSM-VibX0Utr8GCwA4vPXhR4v58if8bZXYLyHkqcaL8mPj0EOfvCoRWmLQKkYYJxE0qndT3zrbrFXKVxvCpdhbQHzy0JwyFPC4kT4BLQFFiEEILztl8mSupXewzaCKpQ"
  }
}
2019-02-20 18:55:43,634:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/reg/51894479:
{
  "protected": "eyJub25jZSI6ICJKRHNLT2E0VldOSURHcWE0NE05OGdmMXEyRGxLQk8weUMySkJkRzBJSWVVIiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAic0RtV2x1VXlhbWtHWlRmazd2QXFuQkxERzI5VzA0RHBuTjBYVWV4M3VZeU9rTnZGYlBBSVFZdzdQbHlaN3BXQTRkbU1Fd2RvblNfNUVOcHVOZ2VLV05NV2tMaWgxaEl5Z0s3czlmak1IOEFPeV9ma1RheV9BeV84RWlSTW4xakJjZlRFYXh5c3ZwdDZOdWFuTE12Mmt6M3cxaExTQWtOU3dRZ245dWZndHBNM01mSm1kQlZQRE5wWGFxRDJuQ0VBRVExSndqSWdZV0R5T2lBNmlGTC1QQVZTTS1WaWJYMFV0cjhHQ3dBNHZQWGhSNHY1OGlmOGJaWFlMeUhrcWNhTDhtUGowRU9mdkNvUldtTFFLa1lZSnhFMHFuZFQzenJickZYS1Z4dkNwZGhiUUh6eTBKd3lGUEM0a1Q0QkxRRkZpRUVJTHp0bDhtU3VwWGV3emFDS3BRIn19",
  "payload": "ewogICJzdGF0dXMiOiAidmFsaWQiLCAKICAiY29udGFjdCI6IFsKICAgICJtYWlsdG86YXJrLnBob25lQGdtYWlsLmNvbSIKICBdLCAKICAicmVzb3VyY2UiOiAicmVnIiwgCiAgImFncmVlbWVudCI6ICJodHRwczovL2xldHNlbmNyeXB0Lm9yZy9kb2N1bWVudHMvTEUtU0EtdjEuMi1Ob3ZlbWJlci0xNS0yMDE3LnBkZiIsIAogICJrZXkiOiB7CiAgICAiZSI6ICJBUUFCIiwgCiAgICAia3R5IjogIlJTQSIsIAogICAgIm4iOiAic0RtV2x1VXlhbWtHWlRmazd2QXFuQkxERzI5VzA0RHBuTjBYVWV4M3VZeU9rTnZGYlBBSVFZdzdQbHlaN3BXQTRkbU1Fd2RvblNfNUVOcHVOZ2VLV05NV2tMaWgxaEl5Z0s3czlmak1IOEFPeV9ma1RheV9BeV84RWlSTW4xakJjZlRFYXh5c3ZwdDZOdWFuTE12Mmt6M3cxaExTQWtOU3dRZ245dWZndHBNM01mSm1kQlZQRE5wWGFxRDJuQ0VBRVExSndqSWdZV0R5T2lBNmlGTC1QQVZTTS1WaWJYMFV0cjhHQ3dBNHZQWGhSNHY1OGlmOGJaWFlMeUhrcWNhTDhtUGowRU9mdkNvUldtTFFLa1lZSnhFMHFuZFQzenJickZYS1Z4dkNwZGhiUUh6eTBKd3lGUEM0a1Q0QkxRRkZpRUVJTHp0bDhtU3VwWGV3emFDS3BRIgogIH0KfQ",
  "signature": "pGpYmM3SSOL1UgO8d1WqBqOh3eNigoZPUhWeqAgjwsimsArQfLUKEgt9djRcIsmzhprY9iJo3AG4GHV-mkPoJ9tOwisZjB-8SyzCILqEb30rC5ViCQAImDNpoM_fyoQlWSE2bxeUn6_PROtb8fwnOAvcZmBea2roSqTinUmoQaST-icb4nMbWVKDGinaH4hxb1k9C3Z0BushqX_GBeySPRqhbkltk5dCiFSOtKLr9Pdj4EYgvfptrsLPvU1-6iYWxWQSAbI_ojx5wlwytLIJUF0OGZGp_yWKpFBeg1BBUO_Et5uPDKTZvndN3V2y_7JW_BtUtZbt20zqONYKb12IHw"
}
2019-02-20 18:55:43,898:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/reg/51894479 HTTP/1.1" 202 654
2019-02-20 18:55:43,900:DEBUG:acme.client:Received response:
HTTP 202
Server: nginx
Content-Type: application/json
Content-Length: 654
Boulder-Requester: 51894479
Link: <https://acme-v01.api.letsencrypt.org/acme/new-authz>;rel="next", <https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel="terms-of-service"
Replay-Nonce: woyI08lDncPswPDQN7wXFniscmD3BbVUiLogTZlNdUQ
Expires: Wed, 20 Feb 2019 18:55:43 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:43 GMT
Connection: keep-alive

{
  "id": 51894479,
  "key": {
    "kty": "RSA",
    "n": "sDmWluUyamkGZTfk7vAqnBLDG29W04DpnN0XUex3uYyOkNvFbPAIQYw7PlyZ7pWA4dmMEwdonS_5ENpuNgeKWNMWkLih1hIygK7s9fjMH8AOy_fkTay_Ay_8EiRMn1jBcfTEaxysvpt6NuanLMv2kz3w1hLSAkNSwQgn9ufgtpM3MfJmdBVPDNpXaqD2nCEAEQ1JwjIgYWDyOiA6iFL-PAVSM-VibX0Utr8GCwA4vPXhR4v58if8bZXYLyHkqcaL8mPj0EOfvCoRWmLQKkYYJxE0qndT3zrbrFXKVxvCpdhbQHzy0JwyFPC4kT4BLQFFiEEILztl8mSupXewzaCKpQ",
    "e": "AQAB"
  },
  "contact": [
    "mailto:<my email>"
  ],
  "agreement": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
  "initialIp": "194.59.165.91",
  "createdAt": "2019-02-20T18:55:43Z",
  "status": "valid"
}
2019-02-20 18:55:43,900:DEBUG:acme.client:Storing nonce: woyI08lDncPswPDQN7wXFniscmD3BbVUiLogTZlNdUQ
2019-02-20 18:55:43,903:DEBUG:certbot.reporter:Reporting to user: Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.
2019-02-20 18:55:43,907:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=u'valid', terms_of_service_agreed=None, agreement=u'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf', only_return_existing=None, contact=(u'mailto:<my email>',), key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f20d0106850>)>)), uri='https://acme-v01.api.letsencrypt.org/acme/reg/51894479', new_authzr_uri=None, terms_of_service='https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'), f206e6e4c087d210bdc64096d2dd619c, Meta(creation_host='056ae666d821', creation_dt=datetime.datetime(2019, 2, 20, 18, 55, 43, tzinfo=<UTC>)))>
2019-02-20 18:55:43,909:INFO:certbot.main:Obtaining a new certificate
2019-02-20 18:55:44,062:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem
2019-02-20 18:55:44,064:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem
2019-02-20 18:55:44,066:DEBUG:acme.client:JWS payload:
{
  "identifier": {
    "type": "dns",
    "value": "<my domain>"
  },
  "resource": "new-authz"
}
2019-02-20 18:55:44,069:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
  "protected": "eyJub25jZSI6ICJ3b3lJMDhsRG5jUHN3UERRTjd3WEZuaXNjbUQzQmJWVWlMb2dUWmxOZFVRIiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAic0RtV2x1VXlhbWtHWlRmazd2QXFuQkxERzI5VzA0RHBuTjBYVWV4M3VZeU9rTnZGYlBBSVFZdzdQbHlaN3BXQTRkbU1Fd2RvblNfNUVOcHVOZ2VLV05NV2tMaWgxaEl5Z0s3czlmak1IOEFPeV9ma1RheV9BeV84RWlSTW4xakJjZlRFYXh5c3ZwdDZOdWFuTE12Mmt6M3cxaExTQWtOU3dRZ245dWZndHBNM01mSm1kQlZQRE5wWGFxRDJuQ0VBRVExSndqSWdZV0R5T2lBNmlGTC1QQVZTTS1WaWJYMFV0cjhHQ3dBNHZQWGhSNHY1OGlmOGJaWFlMeUhrcWNhTDhtUGowRU9mdkNvUldtTFFLa1lZSnhFMHFuZFQzenJickZYS1Z4dkNwZGhiUUh6eTBKd3lGUEM0a1Q0QkxRRkZpRUVJTHp0bDhtU3VwWGV3emFDS3BRIn19",
  "payload": "ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAid3d3LmRhdGFibnBiLm9ubGluZSIKICB9LCAKICAicmVzb3VyY2UiOiAibmV3LWF1dGh6Igp9",
  "signature": "gaALVmb-tTtthDE2nLWVdRR5LCsbyxk2gUNvw2g5oAKkN2aufvYwd1IPTdq9_WCRz2PsX2lTdA7uL1tmXEtEQuwcl3s4Ek6gkGrtaCkiLmEZgtA_7jPK6hWHG2jjTXnM3VNxK0XnKHzieOdc175ThuBTVAWadBQj4OeQ3u4TmaKxrfQSuZ-kTqG0tirCf3U06SECMyfifsLxJNslGhE8NZOZp_EnjQ3eYDoWnrK6h31SKYCdsC7pjsbar7SVe_iYtDZIZWdm0bul21h-1hq4uv7k4KDaV6cuD1j9rvnJ0HhqH2i90VCNHwMLugmriwMpC6QI6NE-h1Mq9FwDZ_yrIQ"
}
2019-02-20 18:55:44,331:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/new-authz HTTP/1.1" 201 1001
2019-02-20 18:55:44,333:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 1001
Boulder-Requester: 51894479
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Location: https://acme-v01.api.letsencrypt.org/acme/authz/<my chalenge code>
Replay-Nonce: oODhJdtJkA4d273llYX2Eraz-jz99fJfB0c1Fcp0zTA
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 20 Feb 2019 18:55:44 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:44 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "<my domain>"
  },
  "status": "pending",
  "expires": "2019-02-27T18:55:44Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316155",
      "token": "FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316156",
      "token": "DoI7O84HBQo5XpTE-cMvJcpsVInze2rlPx2mQJpedTk"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316157",
      "token": "IEwmGJlzyyha_A-QEoyd6Q_hzyD2Q0VKfM8pU4-uR44"
    }
  ],
  "combinations": [
    [
      2
    ],
    [
      1
    ],
    [
      0
    ]
  ]
}
2019-02-20 18:55:44,333:DEBUG:acme.client:Storing nonce: oODhJdtJkA4d273llYX2Eraz-jz99fJfB0c1Fcp0zTA
2019-02-20 18:55:44,337:INFO:certbot.auth_handler:Performing the following challenges:
2019-02-20 18:55:44,338:INFO:certbot.auth_handler:http-01 challenge for <my domain>
2019-02-20 18:55:44,338:DEBUG:acme.standalone:Successfully bound to :1337 using IPv6
2019-02-20 18:55:44,339:DEBUG:acme.standalone:Certbot wasn't able to bind to :1337 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
2019-02-20 18:55:44,350:INFO:certbot.auth_handler:Waiting for verification...
2019-02-20 18:55:44,351:DEBUG:acme.client:JWS payload:
{
  "keyAuthorization": "FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs.iBp7hww56qjcKwQfNa1fPepmvNIy6Ie7y47pQE1Rw6U",
  "type": "http-01",
  "resource": "challenge"
}
2019-02-20 18:55:44,355:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316155:
{
  "protected": "eyJub25jZSI6ICJvT0RoSmR0SmtBNGQyNzNsbFlYMkVyYXotano5OWZKZkIwYzFGY3AwelRBIiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAic0RtV2x1VXlhbWtHWlRmazd2QXFuQkxERzI5VzA0RHBuTjBYVWV4M3VZeU9rTnZGYlBBSVFZdzdQbHlaN3BXQTRkbU1Fd2RvblNfNUVOcHVOZ2VLV05NV2tMaWgxaEl5Z0s3czlmak1IOEFPeV9ma1RheV9BeV84RWlSTW4xakJjZlRFYXh5c3ZwdDZOdWFuTE12Mmt6M3cxaExTQWtOU3dRZ245dWZndHBNM01mSm1kQlZQRE5wWGFxRDJuQ0VBRVExSndqSWdZV0R5T2lBNmlGTC1QQVZTTS1WaWJYMFV0cjhHQ3dBNHZQWGhSNHY1OGlmOGJaWFlMeUhrcWNhTDhtUGowRU9mdkNvUldtTFFLa1lZSnhFMHFuZFQzenJickZYS1Z4dkNwZGhiUUh6eTBKd3lGUEM0a1Q0QkxRRkZpRUVJTHp0bDhtU3VwWGV3emFDS3BRIn19",
  "payload": "ewogICJrZXlBdXRob3JpemF0aW9uIjogIkZWZUxTX2tRMFQzUkZRU2VQNlQzMWFPZXI0MFI2ZUtrV3I1R3k1YTRVRnMuaUJwN2h3dzU2cWpjS3dRZk5hMWZQZXBtdk5JeTZJZTd5NDdwUUUxUnc2VSIsIAogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9",
  "signature": "B8nhV-YPuSezQt-jBByxwneecMLmgV25z-A9snbDqepMcDJsKg7xYNoiPI3_r2fsqlZDIX3UyHitnpzQJXra8_DSZ1sn4J9amLsFo709tU5hOIpyxWWGw0CtouPCMKoEIedmQlNykMIDPllIQ7FO7TXEclsArdUnZwX01ADFJMX9QVS-acbW3KulYHzpl6SxnivxadEg1EcP4Oc_h2CNI7GaMAcTZNApC_vejGoDNDo1rWHozSfiXgjEfqzCOOCASORGxcW-pzFUXEIgjlO6bpm2YOGmttL0dE5CZXSpWUqMrVoWLzW4QPhKSvlmDhUlglOApLzHshhdaNa3M4JX8Q"
}
2019-02-20 18:55:44,619:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/challenge/<my chalenge code>/12806316155 HTTP/1.1" 202 337
2019-02-20 18:55:44,620:DEBUG:acme.client:Received response:
HTTP 202
Server: nginx
Content-Type: application/json
Content-Length: 337
Boulder-Requester: 51894479
Link: <https://acme-v01.api.letsencrypt.org/acme/authz/<my chalenge code>>;rel="up"
Location: https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316155
Replay-Nonce: KIrCIgplScWKuEHejSe4EfU_vme7ieqHH_AaDrkYSHE
Expires: Wed, 20 Feb 2019 18:55:44 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:44 GMT
Connection: keep-alive

{
  "type": "http-01",
  "status": "pending",
  "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316155",
  "token": "FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs",
  "keyAuthorization": "FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs.iBp7hww56qjcKwQfNa1fPepmvNIy6Ie7y47pQE1Rw6U"
}
2019-02-20 18:55:44,622:DEBUG:acme.client:Storing nonce: KIrCIgplScWKuEHejSe4EfU_vme7ieqHH_AaDrkYSHE
2019-02-20 18:55:47,626:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/acme/authz/<my chalenge code>.
2019-02-20 18:55:47,880:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /acme/authz/<my chalenge code> HTTP/1.1" 200 2367
2019-02-20 18:55:47,881:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Replay-Nonce: T_CZzmsB6w9Jh8BFs-LwRYNUoG-NVnPNtStSQYeiq_0
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Content-Length: 2367
Expires: Wed, 20 Feb 2019 18:55:47 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 20 Feb 2019 18:55:47 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "<my domain>"
  },
  "status": "invalid",
  "expires": "2019-02-27T18:55:44Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:unauthorized",
        "detail": "Invalid response from http://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs: \"\u003c!DOCTYPE html\u003e\\n\u003c!--[if lt IE 7]\u003e \u003chtml class=\\\"no-js ie6 oldie\\\" lang=\\\"en-US\\\"\u003e \u003c![endif]--\u003e\\n\u003c!--[if IE 7]\u003e    \u003chtml class=\\\"no-js \"",
        "status": 403
      },
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316155",
      "token": "FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs",
      "validationRecord": [
        {
          "url": "http://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs",
          "hostname": "<my domain>",
          "port": "80",
          "addressesResolved": [
            "104.27.178.114",
            "104.27.179.114",
            "2606:4700:30::681b:b372",
            "2606:4700:30::681b:b272"
          ],
          "addressUsed": "2606:4700:30::681b:b372"
        },
        {
          "url": "https://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs",
          "hostname": "<my domain>",
          "port": "443",
          "addressesResolved": [
            "104.27.178.114",
            "104.27.179.114",
            "2606:4700:30::681b:b372",
            "2606:4700:30::681b:b272"
          ],
          "addressUsed": "2606:4700:30::681b:b372"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316156",
      "token": "DoI7O84HBQo5XpTE-cMvJcpsVInze2rlPx2mQJpedTk"
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<my chalenge code>/12806316157",
      "token": "IEwmGJlzyyha_A-QEoyd6Q_hzyD2Q0VKfM8pU4-uR44"
    }
  ],
  "combinations": [
    [
      2
    ],
    [
      1
    ],
    [
      0
    ]
  ]
}
2019-02-20 18:55:47,883:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: <my domain>
Type:   unauthorized
Detail: Invalid response from http://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-02-20 18:55:47,914:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. <my domain> (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "

2019-02-20 18:55:47,914:DEBUG:certbot.error_handler:Calling registered functions
2019-02-20 18:55:47,914:INFO:certbot.auth_handler:Cleaning up challenges
2019-02-20 18:55:47,916:DEBUG:certbot.plugins.standalone:Stopping server at :::1337...
2019-02-20 18:55:48,350:ERROR:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 1254, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/local/lib/python2.7/dist-packages/certbot/main.py", line 120, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 391, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 334, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/local/lib/python2.7/dist-packages/certbot/client.py", line 370, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/local/lib/python2.7/dist-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. <my domain> (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://<my domain>/.well-known/acme-challenge/FVeLS_kQ0T3RFQSeP6T31aOer40R6eKkWr5Gy5a4UFs: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "

here's the log from letsencrypt.log, (I don't know if any other sensitive information, I hope I don't forgot to replace them 😄)

issa-tseng commented 5 years ago

hmmmm. the only thing i can see before the failure occurs (when letsencrypt tries to connect to your server) that seems weird is that your server has both IPv6 and IPv4 addresses, and however that is set up certbot was only able to attach to the IPv6 but not the IPv4 address. later on, it claims to be trying IPv6, but my experience with IPv6 in general is that sometimes these sorts of splits can cause problems.

either way, the root problem is that your machine is capable of connecting to letsencrypt and setting up the authorization, but that when letsencrypt tries to connect back to you over a direct IP connection, it is unable to do so. there is a small snippet of HTML it outputs from the failure; i'm not sure where that HTML comes from but it resembles neither the ODK Central HTML nor the nginx error page, so i'm not really sure what letsencrypt is hitting instead of the certbot verifier.

arkhub commented 5 years ago

Interesting, didn't though about it before because not long ago letsencrypt in odk central works. It might be my vps provider or cloudflare is the culprit, I'll have a look at that. Meanwhile I'm quite satisfied that customssl using cloudflare works for this time.

issa-tseng commented 5 years ago

okay, great. i'm glad it's working, and thanks for taking a look!