processone / ejabberd

Robust, Ubiquitous and Massively Scalable Messaging Platform (XMPP, MQTT, SIP Server)
https://www.process-one.net/en/ejabberd/
Other
6k stars 1.5k forks source link

ACME fails with IPv6 only ACME server #4170

Closed regalialong closed 2 months ago

regalialong commented 4 months ago

Environment

Configuration (only if needed): grep -Ev '^$|^\s*#' ejabberd.yml

```yaml hosts: - xmpp.demik loglevel: debug listen: - port: 5222 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5223 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s tls: true - port: 5269 ip: "::" module: ejabberd_s2s_in max_stanza_size: 524288 shaper: s2s_shaper - port: 443 ip: "::" module: ejabberd_http tls: true request_handlers: /admin: ejabberd_web_admin /api: mod_http_api /bosh: mod_bosh /captcha: ejabberd_captcha /upload: mod_http_upload /ws: ejabberd_http_ws - port: 80 ip: "::" module: ejabberd_http request_handlers: /admin: ejabberd_web_admin /.well-known/acme-challenge: ejabberd_acme - port: 3478 ip: "::" transport: udp module: ejabberd_stun use_turn: true - port: 1883 ip: "::" module: mod_mqtt acme: ca_url: https://pki.demik/v1/pki_int/acme/directory auto: true s2s_use_starttls: optional acl: local: user_regexp: "" loopback: ip: - 127.0.0.0/8 - ::1/128 access_rules: local: allow: local c2s: deny: blocked allow: all announce: allow: admin configure: allow: admin muc_create: allow: local pubsub_createnode: allow: local trusted_network: allow: loopback api_permissions: "console commands": from: - ejabberd_ctl who: all what: "*" "admin access": who: access: allow: - acl: loopback - acl: admin oauth: scope: "ejabberd:admin" access: allow: - acl: loopback - acl: admin what: - "*" - "!stop" - "!start" "public commands": who: ip: 127.0.0.1/8 what: - status - connected_users_number shaper: normal: rate: 3000 burst_size: 20000 fast: 100000 shaper_rules: max_user_sessions: 10 max_user_offline_messages: 5000: admin 100: all c2s_shaper: none: admin normal: all s2s_shaper: fast modules: mod_adhoc: {} mod_admin_extra: {} mod_announce: access: announce mod_avatar: {} mod_blocking: {} mod_bosh: {} mod_caps: {} mod_carboncopy: {} mod_client_state: {} mod_configure: {} mod_disco: {} mod_fail2ban: {} mod_http_api: {} mod_http_upload: put_url: https://@HOST@:5443/upload custom_headers: "Access-Control-Allow-Origin": "https://@HOST@" "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS" "Access-Control-Allow-Headers": "Content-Type" mod_last: {} mod_mam: assume_mam_usage: true default: always mod_mqtt: {} mod_muc: access: - allow access_admin: - allow: admin access_create: muc_create access_persistent: muc_create access_mam: - allow default_room_options: mam: true mod_muc_admin: {} mod_offline: access_max_user_messages: max_user_offline_messages mod_ping: {} mod_privacy: {} mod_private: {} mod_pubsub: access_createnode: pubsub_createnode plugins: - flat - pep force_node_config: storage:bookmarks: access_model: whitelist mod_push: {} mod_push_keepalive: {} mod_register: ip_access: trusted_network mod_roster: versioning: true mod_s2s_dialback: {} mod_shared_roster: {} mod_stream_mgmt: resend_on_timeout: if_offline mod_stun_disco: {} mod_vcard: {} mod_vcard_xupdate: {} mod_version: show_os: false ```

Errors from error.log/crash.log

2024-03-07 03:33:44.462554+00:00 [error] <0.739.0>@ejabberd_acme:issue_request/7:246
 Failed to request certificate for xmpp.demik, pubsub.xmpp.demik and 2 more hosts:
 HTTP error: transport failure: non-existing domain

Bug description

I have a IPv6-only VPN with internal TLD and I wanted to try out ejabberd. I have an ACME server at https://pki.demik which I want to get the certificates for the XMPP server for but ejabberd fails to connect to it with non-existing domain which doesn't make sense because the rest of the system resolves and connects fine.

Checking logs on debug loglevel shows the connection to the ACME server fails:

Mar 07 04:07:57 xmpp ejabberdctl[31524]: 2024-03-07 04:07:57.563622+00:00 [error] Failed to request certificate for xmpp.demik, pubsub.xmpp.demik and 2 more hosts: HTTP error: transport failure: non-existing domain
Mar 07 04:07:57 xmpp ejabberdctl[31524]:                                        {inet,[inet],nxdomain}]}}
Mar 07 04:07:57 xmpp ejabberdctl[31524]: 2024-03-07 04:07:57.563491+00:00 [debug] HTTP response: {error,{failed_connect,[{to_address,{"pki.demik",443}},
Mar 07 04:07:57 xmpp ejabberdctl[31524]: 2024-03-07 04:07:57.477251+00:00 [debug] HTTP request: {get,{"https://pki.demik/v1/pki_int/acme/directory",[]}}

Trying to diagnose this in the debug shell, inet_res:getbyname("pki.demik", aaaa). fails which is fixed adding {inet6, true}. to inetrc. ACME still fails with non-existing domain though.

I think this is happening because httpc is at the default of inet for ipfamily instead of inet6, connections to ipv6.google.com similarly fail with nxdomain.

(ejabberd@localhost)2>  httpc:request("https://ipv6.google.com").
{error,{failed_connect,[{to_address,{"ipv6.google.com",443}},
                        {inet,[inet],nxdomain}]}}
(ejabberd@localhost)3> 

Setting inet6 makes the same request work:

(ejabberd@localhost)1> httpc:set_options([{ipfamily, inet6}]).
ok
(ejabberd@localhost)2> httpc:request("https://ipv6.google.com").
{ok,{{"HTTP/1.1",200,"OK"},
...

And also makes the connection to the ACME server work with ejabberdctl request_certificate all:

Mar 07 04:19:00 xmpp ejabberdctl[33884]: 2024-03-07 04:19:00.867889+00:00 [info] Certificate for xmpp.demik, pubsub.xmpp.demik and 2 more hosts has been received, stored and loaded successfully

Is this an oversight or is this just an unsupported usecase?

bestperson-free commented 3 months ago

Try adding contact, and I also do this iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 5280 && ejabberdctl request-certificate all && ejabberdctl list-certificates && iptables -t nat -D PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 5280

acme:
  ## Staging environment
  contact: mailto:post@vivaldi.net
#  ca_url: https://acme-staging-v02.api.letsencrypt.org/directory
#  auto: false
  ## Production environment (the default):
  ca_url: https://acme-v02.api.letsencrypt.org/directory
  auto: false
#  ca_url: https://acme-v02.api.letsencrypt.org
``` yaml ### ### ejabberd configuration file ### ### The parameters used in this configuration file are explained at ### ### https://docs.ejabberd.im/admin/configuration ### ### The configuration file is written in YAML. ### ******************************************************* ### ******* !!! WARNING !!! ******* ### ******* YAML IS INDENTATION SENSITIVE ******* ### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY ******* ### ******************************************************* ### Refer to http://en.wikipedia.org/wiki/YAML for the brief description. ### hosts: - jabber.doman.ru loglevel: error ca_file: /opt/ejabberd/conf/cacert.pem certfiles: - /opt/ejabberd/conf/server.pem ## If you already have certificates, list them here # certfiles: # - /opt/ejabberd/certs/jabber.doman.ru.fullchain.pem # - /opt/ejabberd/certs/jabber.doman.ru.privkey.pem # - /opt/ejabberd/certs/upload.jabber.doman.ru.fullchain.pem # - /opt/ejabberd/certs/upload.jabber.doman.ru.privkey.pem # - /opt/ejabberd/certs/conference.jabber.doman.ru.fullchain.pem # - /opt/ejabberd/certs/conference.jabber.doman.ru.privkey.pem # - /opt/ejabberd/certs/pubsub.jabber.doman.ru.fullchain.pem # - /opt/ejabberd/certs/pubsub.jabber.doman.ru.privkey.pem # - /opt/ejabberd/certs/proxy.jabber.doman.ru.fullchain.pem # - /opt/ejabberd/certs/proxy.jabber.doman.ru.privkey.pem listen: - port: 5222 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5223 ip: "::" tls: true module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5269 ip: "::" module: ejabberd_s2s_in max_stanza_size: 524288 - port: 5443 ip: "::" module: ejabberd_http tls: true request_handlers: # /admin: ejabberd_web_admin /api: mod_http_api /bosh: mod_bosh # /captcha: ejabberd_captcha /upload: mod_http_upload /ws: ejabberd_http_ws - port: 5280 ip: "::" module: ejabberd_http tls: false request_handlers: # /admin: ejabberd_web_admin /.well-known/acme-challenge: ejabberd_acme - port: 3478 transport: udp module: ejabberd_stun use_turn: true turn_min_port: 49152 turn_max_port: 65535 ## The server's public IPv4 address: turn_ipv4_address: 188.225.72.69 ## The server's public IPv6 address: # turn_ipv6_address: "2001:db8::3" - port: 5349 transport: tcp module: ejabberd_stun use_turn: true tls: true turn_min_port: 49152 turn_max_port: 65535 ip: "::" ## The server's public IPv4 address: turn_ipv4_address: 188.225.72.69 # - # port: 1883 # ip: "::" # module: mod_mqtt # backlog: 1000 acme: ## Staging environment contact: mailto:post@vivaldi.net # ca_url: https://acme-staging-v02.api.letsencrypt.org/directory # auto: false ## Production environment (the default): ca_url: https://acme-v02.api.letsencrypt.org/directory auto: false # ca_url: https://acme-v02.api.letsencrypt.org s2s_use_starttls: required acl: admin: - user: user@jabber.doman.ru local: user_regexp: "" loopback: ip: - 127.0.0.0/8 - ::1/128 - ::FFFF:127.0.0.1/128 access_rules: local: allow: local c2s: deny: blocked allow: all announce: allow: admin configure: allow: admin muc_create: allow: local pubsub_createnode: allow: local trusted_network: allow: loopback api_permissions: "console commands": from: - ejabberd_ctl who: all what: "*" "admin access": who: access: allow: - acl: loopback - acl: admin oauth: scope: "ejabberd:admin" access: allow: - acl: loopback - acl: admin what: - "*" - "!stop" - "!start" "public commands": who: ip: 127.0.0.1/8 what: - status - connected_users_number shaper: normal: rate: 3000 burst_size: 20000 fast: 100000 shaper_rules: max_user_sessions: 10 max_user_offline_messages: 5000: admin 100: all c2s_shaper: none: admin normal: all s2s_shaper: fast modules: mod_adhoc: {} mod_admin_extra: {} mod_announce: access: announce mod_avatar: {} mod_blocking: {} mod_bosh: {} mod_caps: {} mod_carboncopy: {} mod_client_state: {} mod_configure: {} mod_disco: server_info: - modules: all name: "abuse-addresses" urls: ["mailto:post@vivaldi.net"] # - # modules: # - mod_disco # - mod_vcard # name: admin-addresses # urls: # - https://example.com # - mailto:admin@example.com # - xmpp:admin@example.com mod_fail2ban: {} mod_http_api: {} mod_http_upload: put_url: https://@HOST@:5443/upload max_size: 262144000 #250mb custom_headers: "Access-Control-Allow-Origin": "https://@HOST@" "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS" "Access-Control-Allow-Headers": "Content-Type" mod_http_upload_quota: max_days: 90 mod_last: {} mod_mam: ## Mnesia is limited to 2GB, better to use an SQL backend ## For small servers SQLite is a good fit and is very easy ## to configure. Uncomment this when you have SQL configured: ## db_type: sql assume_mam_usage: true default: always mod_mqtt: {} mod_muc: access: - allow access_admin: - allow: admin access_create: muc_create access_persistent: muc_create access_mam: - allow default_room_options: allow_subscription: true persistent: true mam: true mod_muc_admin: {} mod_offline: access_max_user_messages: max_user_offline_messages mod_ping: {} mod_privacy: {} mod_private: {} # mod_proxy65: # port: 2053 # ip: 31.129.105.100 # access: local # max_connections: 5 mod_pubsub: access_createnode: pubsub_createnode plugins: - flat - pep force_node_config: ## Avoid buggy clients to make their bookmarks public storage:bookmarks: access_model: whitelist mod_push: {} mod_push_keepalive: {} mod_register: ## Only accept registration requests from the "trusted" ## network (see access_rules section above). ## Think twice before enabling registration from any ## address. See the Jabber SPAM Manifesto for details: ## https://github.com/ge0rg/jabber-spam-fighting-manifesto ip_access: trusted_network mod_roster: versioning: true mod_s2s_dialback: {} mod_shared_roster: {} mod_stream_mgmt: resend_on_timeout: if_offline mod_stun_disco: credentials_lifetime: 12h services: - host: 188.225.72.69 port: 3478 type: stun transport: udp restricted: false - host: 188.225.72.69 port: 3478 type: turn transport: udp restricted: true - host: jabber.doman.ru port: 5349 type: stuns transport: tcp restricted: false - host: jabber.doman.ru port: 5349 type: turns transport: tcp restricted: true mod_vcard: {} mod_vcard_xupdate: {} mod_version: show_os: false #update_sql_schema: true default_db: sql auth_method: sql #MariaDB with ejabberd sql_type: mysql sql_server: localhost sql_database: admin sql_username: admin sql_password: admin ## If you want to specify the port: sql_port: 3306 language: ru ### Local Variables: ### mode: yaml ### End: ### vim: set filetype=yaml tabstop=8 ```
regalialong commented 3 months ago

I added contact but this doesn't fix it. You don't have this issue, assuming the configuration file you linked is what you are using, you are using Let's Encrypt's endpoints which provide access over IPv4, the internal ACME server of mine does not have an IPv4 endpoint and fails to connect. Contact missing would not cause nxdomain.

regalialong commented 2 months ago

I hate doing this but bump this, I think the fix is practically just setting httpc:set_options([{ipfamily, inet6}]). This is really showstopping ejabberd for us.

badlop commented 2 months ago

httpc:set_options([{ipfamily, inet6}]).

But if this is used when the server is IPv4-only, the query will fail, right?


Looking at the documentation, I found that the ipfamily option supports inet6fb4 since at leat Erlang 26:

With inet6fb4 option, IPv6 will be preferred but if connection fails, an IPv4 fallback connection attempt will be made.

In Erlang/OTP 26 and 27.0-rc2 it is documented:

IpFamily = inet | inet6 | local | inet6fb4

Using Erlang 27.0-rc2:

$ erl -s inets -s ssl
Erlang/OTP 27 [erts-14.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Eshell V14.3 (press Ctrl+G to abort, type help(). for help)
1> httpc:set_options([{ipfamily, inet6fb4}]).
ok
2> httpc:request("https://ipv6.google.com").
{error,{failed_connect,[{to_address,{"ipv6.google.com",443}},
                        {inet6,[inet6],enetunreach},
                        {inet,[inet],nxdomain}]}}

In Erlang/OTP 25.0 Release Candidate 3 that option was not yet documented:

IpFamily = inet | inet6 | local (unix socket)

However, looking at the Erlang source code, I suspect that this option is supported since many more versions ago (at least since Erlang 20.3), even if it was not documented:

Erlang/OTP 20 [erts-9.3.3.15] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3.3.15  (abort with ^G)
1> ssl:start().
ok
2> inets:start().
ok
3>  httpc:set_options([{ipfamily, inet6fb4}]).
ok
4>  httpc:request("https://ipv6.google.com").
{error,{failed_connect,[{to_address,{"ipv6.google.com",443}},
                        {inet6,[inet6],enetunreach},
                        {inet,[inet],nxdomain}]}}

Assuming that the ipv6fb4 is available and works correctly since at least Erlang 20 (this is the oldest version supported by ejabberd and p1_acme), then a patch like this could be safely added to p1_acme:

diff --git a/src/p1_acme.erl b/src/p1_acme.erl
index b80f539..8e53502 100644
--- a/src/p1_acme.erl
+++ b/src/p1_acme.erl
@@ -598,6 +598,7 @@ http_request(State, ReqFun, RetryTimeout) ->
    Timeout ->
        {Method, URL} = Request = ReqFun(State),
        ?DEBUG("HTTP request: ~p", [Request]),
+            httpc:set_options([{ipfamily, inet6fb4}]),
        case httpc:request(Method, URL,
                   [{timeout, infinity},
                {connect_timeout, infinity}],
nosnilmot commented 2 months ago

A better way to test if inet6fb4 does not break IPv4-only uses is to try connecting to an IPv4-only host:

> httpc:set_options([{ipfamily, inet6}]).
ok
> httpc:request("https://ipv4.google.com").
{error,{failed_connect,[{to_address,{"ipv4.google.com",443}},
                        {inet6,[inet6],nxdomain}]}}
> httpc:set_options([{ipfamily, inet6fb4}]).
ok
> httpc:request("https://ipv4.google.com").
{ok,{{"HTTP/1.1",200,"OK"},

a patch like this could be safely added to p1_acme

I would suggest this (untested) which only sets the options once and doesn't interfere with global/default httpc profile:

diff --git a/src/p1_acme.erl b/src/p1_acme.erl
index b80f539..a648bea 100644
--- a/src/p1_acme.erl
+++ b/src/p1_acme.erl
@@ -113,6 +113,9 @@
 %%% API
 %%%===================================================================
 start() ->
+    application:start(inets),
+    inets:start(httpc, [{profile, p1_acme}]),
+    httpc:set_options([{ip_family, inet6fb4}], p1_acme),
     case application:ensure_all_started(?MODULE) of
        {ok, _} -> ok;
        Err -> Err
@@ -602,7 +605,7 @@ http_request(State, ReqFun, RetryTimeout) ->
                               [{timeout, infinity},
                                {connect_timeout, infinity}],
                               [{body_format, binary},
-                               {sync, false}]) of
+                               {sync, false}], p1_acme) of
                {ok, Ref} ->
                    ReqTimeout = min(timer:seconds(10), Timeout),
                    receive
@@ -612,7 +615,7 @@ http_request(State, ReqFun, RetryTimeout) ->
                              ReqFun, Response, State, RetryTimeout)
                    after ReqTimeout ->
                            ?DEBUG("HTTP request timeout", []),
-                           httpc:cancel_request(Ref),
+                           httpc:cancel_request(Ref, p1_acme),
                            http_request(State, ReqFun, RetryTimeout)
                    end;
                {error, WTF} ->
badlop commented 2 months ago

Thanks! I've tested your patch and seems to work, so I've applied it to p1_acme, and updated ejabberd to use it.

It would be great if somebody else can test the feature and report the results, @regalialong ;)

There are binary installers and the container image.

regalialong commented 2 months ago

I pulled the deb package from the actions run you have linked and ejabberd_acme appears to crash for me:

``` Apr 22 19:34:42 xmpp ejabberdctl[61414]: {ssl, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {http_options,"HTTP/1.1",infinity,true, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {[],[]}, Apr 22 19:34:42 xmpp ejabberdctl[61414]: undefined,undefined,[]}, Apr 22 19:34:42 xmpp ejabberdctl[61414]: undefined,undefined,"0",undefined,undefined,undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: undefined,undefined,undefined,undefined,[],undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: "pki.demik",undefined,undefined,undefined,undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: undefined,undefined,undefined,undefined,undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: undefined,undefined,undefined,undefined,undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {http_request_h,undefined,"keep-alive",undefined,undefined, Apr 22 19:34:42 xmpp ejabberdctl[61414]: "/v1/pki_int/acme/directory",[],get, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {"pki.demik",443}, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {request,undefined,<0.742.0>,0,https, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {request, Apr 22 19:34:42 xmpp ejabberdctl[61414]: [httpc_p1_acme, Apr 22 19:34:42 xmpp ejabberdctl[61414]: {gen_server,call, Apr 22 19:34:42 xmpp ejabberdctl[61414]: ** {{noproc, Apr 22 19:34:42 xmpp ejabberdctl[61414]: ** Reason for termination == Apr 22 19:34:42 xmpp ejabberdctl[61414]: {state}] Apr 22 19:34:42 xmpp ejabberdctl[61414]: {'$ancestors',[ejabberd_sup,<0.189.0>]}], Apr 22 19:34:42 xmpp ejabberdctl[61414]: ** When Server state == [[{'$initial_call',{ejabberd_acme,init,1}}, Apr 22 19:34:42 xmpp ejabberdctl[61414]: ** Last message in was {'$gen_cast',ejabberd_started} Apr 22 19:34:42 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.909624+00:00 [error] ** Generic server ejabberd_acme terminating Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.828746+00:00 [warning] Option 'turn_ipv4_address' is undefined and the server's hostname doesn't resolve to a public IPv4 address, most likely the TURN relay won't be working> Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826209+00:00 [info] Start accepting TCP connections at [::]:5269 for ejabberd_s2s_in Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826199+00:00 [info] Start accepting TLS connections at [::]:5223 for ejabberd_c2s Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826268+00:00 [info] Start accepting TCP connections at 127.0.1.1:7777 for mod_proxy65_stream Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826239+00:00 [info] Start accepting TCP connections at [::]:1883 for mod_mqtt Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826215+00:00 [info] Start accepting TCP connections at [::]:80 for ejabberd_http Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826208+00:00 [info] Start accepting TLS connections at [::]:5443 for ejabberd_http Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826167+00:00 [info] Start accepting UDP connections at [::]:3478 for ejabberd_stun Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.826075+00:00 [info] Start accepting TCP connections at [::]:5222 for ejabberd_c2s Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.825934+00:00 [info] Requesting new certificate for xmpp.demik, pubsub.xmpp.demik and 3 more hosts from https://pki.demik/v1/pki_int/acme/directory Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.825811+00:00 [info] ejabberd 24.2.62 is started in the node ejabberd@localhost in 0.80s Apr 22 19:34:41 xmpp sudo[61155]: pam_unix(sudo:session): session closed for user root Apr 22 19:34:41 xmpp systemd[1]: Started ejabberd.service - XMPP Server. Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.821606+00:00 [warning] No certificate found matching proxy.xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.821546+00:00 [warning] No certificate found matching upload.xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.821493+00:00 [warning] No certificate found matching conference.xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.821043+00:00 [warning] No certificate found matching pubsub.xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.820902+00:00 [warning] No certificate found matching xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.797113+00:00 [warning] Invalid certificate in /opt/ejabberd/conf/server.pem: at line 53: self-signed certificate Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.792612+00:00 [info] Waiting for Mnesia synchronization to complete Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.763576+00:00 [warning] The option captcha_cmd is not configured, but some module wants to use the CAPTCHA feature. Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.757733+00:00 [info] Building MQTT cache for xmpp.demik, this may take a while Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.677801+00:00 [info] Won't auto-announce STUN/TURN service on port 3478 (udp) without public IP address, please specify 'turn_ipv4_address' and optionally 'turn_ipv6_address' Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.598934+00:00 [warning] Mnesia backend for mod_mam is not recommended: it's limited to 2GB and often gets corrupted when reaching this limit. SQL backend is recommended. Namel> Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.596142+00:00 [info] Loading modules for xmpp.demik Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.511593+00:00 [info] Watchdog notifications enabled Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.391026+00:00 [info] Configuration loaded successfully Apr 22 19:34:41 xmpp ejabberdctl[61414]: 2024-04-22 19:34:41.121909+00:00 [info] Loading configuration from /opt/ejabberd/conf/ejabberd.yml Apr 22 19:34:40 xmpp ejabberdctl[61414]: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by runni> Apr 22 19:34:40 xmpp systemd[1]: Starting ejabberd.service - XMPP Server... ```
``` yaml ### ### ejabberd configuration file ### ### The parameters used in this configuration file are explained at ### ### https://docs.ejabberd.im/admin/configuration ### ### The configuration file is written in YAML. ### ******************************************************* ### ******* !!! WARNING !!! ******* ### ******* YAML IS INDENTATION SENSITIVE ******* ### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY ******* ### ******************************************************* ### Refer to http://en.wikipedia.org/wiki/YAML for the brief description. ### hosts: - xmpp.demik loglevel: info ca_file: /usr/local/share/ca-certificates/DEMik/rootCA.crt certfiles: - /opt/ejabberd/conf/server.pem ## If you already have certificates, list them here # certfiles: # - /etc/letsencrypt/live/domain.tld/fullchain.pem # - /etc/letsencrypt/live/domain.tld/privkey.pem acme: ca_url: https://pki.demik/v1/pki_int/acme/directory auto: true listen: - port: 5222 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s starttls_required: true - port: 5223 ip: "::" module: ejabberd_c2s max_stanza_size: 262144 shaper: c2s_shaper access: c2s tls: true - port: 5269 ip: "::" module: ejabberd_s2s_in max_stanza_size: 524288 shaper: s2s_shaper - port: 5443 ip: "::" module: ejabberd_http tls: true request_handlers: /admin: ejabberd_web_admin /api: mod_http_api /bosh: mod_bosh /captcha: ejabberd_captcha /upload: mod_http_upload /ws: ejabberd_http_ws - port: 80 ip: "::" module: ejabberd_http request_handlers: /admin: ejabberd_web_admin /.well-known/acme-challenge: ejabberd_acme - port: 3478 ip: "::" transport: udp module: ejabberd_stun use_turn: true ## The server's public IPv4 address: # turn_ipv4_address: "203.0.113.3" ## The server's public IPv6 address: # turn_ipv6_address: "2001:db8::3" - port: 1883 ip: "::" module: mod_mqtt backlog: 1000 s2s_use_starttls: optional acl: local: user_regexp: "" loopback: ip: - 127.0.0.0/8 - ::1/128 access_rules: local: allow: local c2s: deny: blocked allow: all announce: allow: admin configure: allow: admin muc_create: allow: local pubsub_createnode: allow: local trusted_network: allow: loopback api_permissions: "console commands": from: - ejabberd_ctl who: all what: "*" "admin access": who: access: allow: - acl: loopback - acl: admin oauth: scope: "ejabberd:admin" access: allow: - acl: loopback - acl: admin what: - "*" - "!stop" - "!start" "public commands": who: ip: 127.0.0.1/8 what: - status - connected_users_number shaper: normal: rate: 3000 burst_size: 20000 fast: 100000 shaper_rules: max_user_sessions: 10 max_user_offline_messages: 5000: admin 100: all c2s_shaper: none: admin normal: all s2s_shaper: fast modules: mod_adhoc: {} mod_admin_extra: {} mod_announce: access: announce mod_avatar: {} mod_blocking: {} mod_bosh: {} mod_caps: {} mod_carboncopy: {} mod_client_state: {} mod_configure: {} mod_disco: {} mod_fail2ban: {} mod_http_api: {} mod_http_upload: put_url: https://@HOST@:5443/upload custom_headers: "Access-Control-Allow-Origin": "https://@HOST@" "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS" "Access-Control-Allow-Headers": "Content-Type" mod_last: {} mod_mam: ## Mnesia is limited to 2GB, better to use an SQL backend ## For small servers SQLite is a good fit and is very easy ## to configure. Uncomment this when you have SQL configured: ## db_type: sql assume_mam_usage: true default: always mod_mqtt: {} mod_muc: access: - allow access_admin: - allow: admin access_create: muc_create access_persistent: muc_create access_mam: - allow default_room_options: mam: true mod_muc_admin: {} mod_offline: access_max_user_messages: max_user_offline_messages mod_ping: {} mod_privacy: {} mod_private: {} mod_proxy65: access: local max_connections: 5 mod_pubsub: access_createnode: pubsub_createnode plugins: - flat - pep force_node_config: ## Avoid buggy clients to make their bookmarks public storage:bookmarks: access_model: whitelist mod_push: {} mod_push_keepalive: {} mod_register: ## Only accept registration requests from the "trusted" ## network (see access_rules section above). ## Think twice before enabling registration from any ## address. See the Jabber SPAM Manifesto for details: ## https://github.com/ge0rg/jabber-spam-fighting-manifesto ip_access: trusted_network mod_roster: versioning: true mod_s2s_dialback: {} mod_shared_roster: {} mod_stream_mgmt: resend_on_timeout: if_offline mod_stun_disco: {} mod_vcard: {} mod_vcard_xupdate: {} mod_version: show_os: false ### Local Variables: ### mode: yaml ### End: ### vim: set filetype=yaml tabstop=8 ```

Everything beyond what I sent dumps what looks like entire certificate store. The system does trust our CA certificate (curl and others work), same happens no matter if I specify ca_file or comment it out.

I also ran certbot on the same machine to verify that ACME works and that renews successfully as well.

badlop commented 2 months ago

Aha! I can reproduce the problem, ejabberd crashes on start.

The problem is that the function p1_acme:start is not called, so the httpc_p1_acme process is not started, and that one is required since the recent patch to perform the HTTP queries, as it contains the specific HTTP options.

I've applied an additional patch to p1_acme, and built installers in my ejabberd fork: https://github.com/badlop/ejabberd/actions/runs/8802053312

Now ejabberd starts for me. Can you try that? Once you confirm it works perfectly, I'll merge into ejabberd upstream.

regalialong commented 2 months ago

Apr 23 15:35:07 xmpp ejabberdctl[65391]: 2024-04-23 15:35:07.370501+00:00 [info] Certificate for xmpp.demik, pubsub.xmpp.demik and 3 more hosts has been received, stored and loaded successfully Yep, that works fully now. Thank you very much for the help :)

badlop commented 2 months ago

Ok, ejabberd already points to the latest p1_acme, the next version will include it. Thanks all for the report, patches and testing!