ACME Failed to request certificate 404 (error type: unauthorized)

[APoniatowski]

Before creating a ticket, please consider if this should fit the discussion forum better.

Environment

• ejabberd version: 23.4.0
• Erlang version: erl +V
• OS: Docker
• Installed from: Docker

Configuration (only if needed): grep -Ev '^$|^\s*#' ejabberd.yml

loglevel: 4
...

Errors from error.log/crash.log

ejabberd | 2023-08-09 12:37:45.005087+00:00 [error] Failed to request certificate for DOMAIN.ORG, pubsub.DOMAIN.ORG and 3 more hosts: Challenge failed for domain conference.DOMAIN.ORG: ACME server reported: IP_ADDRESS: Invalid response from http://conference.DOMAIN.ORG/.well-known/acme-challenge/yozIpF6GR9GiZ2qx4WgkjHqd-QkFgW-FiKHdEtHbilE: 404 (error type: unauthorized)

Bug description

getting the following when trying to reach the admin panel: An error occurred during a connection to DOMAIN.ORG:5443. PR_END_OF_FILE_ERROR

Error code: PR_END_OF_FILE_ERROR

I am also using a .dev domain name, if that helps. I hope there is a simple and quick fix for this, as I'd rather use ejabberd over prosody

[licaon-kter]

Did you setup conference. as needed in the webserver?

[APoniatowski]

I'm not sure where to configure this, as I am using the docker image via docker compose.

[APoniatowski]

I am using the example config file, with very slightly modified attributes

```yaml ### ### ejabberd configuration file ### ### The parameters used in this configuration file are explained at ### ### https://docs.ejabberd.im/admin/configuration ### ### The configuration file is written in YAML. ### ******************************************************* ### ******* !!! WARNING !!! ******* ### ******* YAML IS INDENTATION SENSITIVE ******* ### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY ******* ### ******************************************************* ### Refer to http://en.wikipedia.org/wiki/YAML for the brief description. ### hosts: - DOMAIN.ORG loglevel: 4 ## If you already have certificates, list them here # certfiles: # - /etc/letsencrypt/live/domain.tld/fullchain.pem # - /etc/letsencrypt/live/domain.tld/privkey.pem listen: - port: 5222 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5223 ip: "::" tls: true module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5269 ip: "::" module: ejabberd_s2s_in max_stanza_size: 524288 - port: 5443 ip: "::" module: ejabberd_http tls: true request_handlers: /admin: ejabberd_web_admin /api: mod_http_api /bosh: mod_bosh /captcha: ejabberd_captcha /upload: mod_http_upload /ws: ejabberd_http_ws - port: 5280 ip: "::" module: ejabberd_http tls: false request_handlers: /.well-known/acme-challenge: ejabberd_acme - port: 3478 ip: "::" transport: udp module: ejabberd_stun use_turn: true ## The server's public IPv4 address: # turn_ipv4_address: "203.0.113.3" ## The server's public IPv6 address: # turn_ipv6_address: "2001:db8::3" - port: 1883 ip: "::" module: mod_mqtt backlog: 1000 s2s_use_starttls: optional acl: local: user_regexp: "" loopback: ip: - 127.0.0.0/8 - ::1/128 access_rules: local: allow: local c2s: deny: blocked allow: all announce: allow: admin configure: allow: admin muc_create: allow: local pubsub_createnode: allow: local trusted_network: allow: loopback api_permissions: "console commands": from: - ejabberd_ctl who: all what: "*" "admin access": who: access: allow: - acl: loopback - acl: admin oauth: scope: "ejabberd:admin" access: allow: - acl: loopback - acl: admin what: - "*" - "!stop" - "!start" "public commands": who: ip: 127.0.0.1/8 what: - status - connected_users_number shaper: normal: rate: 3000 burst_size: 20000 fast: 100000 shaper_rules: max_user_sessions: 10 max_user_offline_messages: 5000: admin 100: all c2s_shaper: none: admin normal: all s2s_shaper: fast modules: mod_adhoc: {} mod_admin_extra: {} mod_announce: access: announce mod_avatar: {} mod_blocking: {} mod_bosh: {} mod_caps: {} mod_carboncopy: {} mod_client_state: {} mod_configure: {} mod_disco: {} mod_fail2ban: {} mod_http_api: {} mod_http_upload: put_url: https://@HOST@:5443/upload custom_headers: "Access-Control-Allow-Origin": "https://@HOST@" "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS" "Access-Control-Allow-Headers": "Content-Type" mod_last: {} mod_mam: ## Mnesia is limited to 2GB, better to use an SQL backend ## For small servers SQLite is a good fit and is very easy ## to configure. Uncomment this when you have SQL configured: ## db_type: sql assume_mam_usage: true default: always mod_mqtt: {} mod_muc: access: - allow access_admin: - allow: admin access_create: muc_create access_persistent: muc_create access_mam: - allow default_room_options: mam: true mod_muc_admin: {} mod_offline: access_max_user_messages: max_user_offline_messages mod_ping: {} mod_privacy: {} mod_private: {} mod_proxy65: access: local max_connections: 5 mod_pubsub: access_createnode: pubsub_createnode plugins: - flat - pep force_node_config: ## Avoid buggy clients to make their bookmarks public storage:bookmarks: access_model: whitelist mod_push: {} mod_push_keepalive: {} mod_register: ## Only accept registration requests from the "trusted" ## network (see access_rules section above). ## Think twice before enabling registration from any ## address. See the Jabber SPAM Manifesto for details: ## https://github.com/ge0rg/jabber-spam-fighting-manifesto ip_access: trusted_network mod_roster: versioning: true mod_s2s_dialback: {} mod_shared_roster: {} mod_stream_mgmt: resend_on_timeout: if_offline mod_stun_disco: {} mod_vcard: {} mod_vcard_xupdate: {} mod_version: show_os: false ### Local Variables: ### mode: yaml ### End: ### vim: set filetype=yaml tabstop=8 ```

[licaon-kter]

You've opened port 80 and 5280?

[prefiks]

Do you have proxy that deliver requests from port 80 on your domain to 5280 on ejabberd? Is that also accepting connections from conference.domain.com?

[APoniatowski]

ports are open (managed by cloud provider firewall and it is open) and not using an external nginx frontend. Unless the docker image has one

[licaon-kter]

Can you list the docker command you use to run?

[APoniatowski]

I use docker compose:

version: '3.7'

services:

  main:
    image: ghcr.io/processone/ejabberd
    container_name: ejabberd
    environment:
      - CTL_ON_CREATE=register admin DOMAIN.ORG SOME_PASS
      - CTL_ON_START=registered_users DOMAIN.ORG ;
                     registered_vhosts ;
                     status
    ports:
      - "5222:5222"
      - "5269:5269"
      - "5280:5280"
      - "5443:5443"
    volumes:
      - ./ejabberd.yml:/opt/ejabberd/conf/ejabberd.yml:ro
      - ./database:/opt/ejabberd/database
      - ./logs:/opt/ejabberd/logs
      - ./upload:/opt/ejabberd/upload

[licaon-kter]

And 80 should redirect to 5280 as said, yes?

[APoniatowski]

I followed the documentation, so no mention of a redirect on the docker image. I could try and change '5280:5208' to '80:5280'? if that is what you meant

[licaon-kter]

The docs say to open port 80 and redirect to 5280 though, your command does not open 80 on the docker, right?

Note that the ACME protocol requires challenges to be sent on port 80. Since this is a privileged port, ejabberd cannot listen on it directly without root privileges. Thus you need some mechanism to forward port 80 to the port defined by the listener (port 5280 in the example above). There are several ways to do this: using NAT, setcap (Linux only), or HTTP front-ends (e.g. sslh, nginx, haproxy and so on). Pick one that fits your installation the best, but DON'T run ejabberd as root.

https://docs.ejabberd.im/admin/configuration/basic/#setting-up-acme

[APoniatowski]

I was using this doc to set it up:

https://github.com/processone/ejabberd/blob/master/CONTAINER.md

So it might need to be updated, as there is no mention of a redirect (I assumed nginx is preinstalled on the container, and configured to use 5280 out of the box, but I was wrong)

I have however check if one of my other services are taking that port, and noticed that it has. So I will need to make some changes to it.

Thanks for the help in finding the root cause of my issue

[licaon-kter]

https://github.com/processone/ejabberd/blob/master/CONTAINER.md

That's the generic intro, there's no mention of ACME there.

Once that is ready you still need to read ejabberd docs: https://docs.ejabberd.im ;)