[bug]: not possible to create Let's Encrypt certificate

nobe80 commented 5 months ago

FreePBX Version

FreePBX 17

Issue Description

We are using FreePBX V17 on Debian 12. All modules are update to the newest edge versions It is not possible to create LE certificate. Doesnt matter with or without enabled firewall. DNS is set correctly as well as Port configuration in FreepBX port management

We get always this output: Processing: bitpbx.beonit-cloud.de, Local IP: 116.203.228.6, Public IP: dns error Self test: trying http://bitpbx.beonit-cloud.de/.freepbx-known/d68b29f743b987218d6795533f24f046 Self test: received d68b29f743b987218d6795533f24f046

Operating Environment

Debian 12.5 FreePBX 17.0.15.15 1 network nic (Local ip == Public IP)

Relevant log output

No response

dolesec commented 5 months ago

@nobe80 ive seen this as well in testing as recently as yesterday ... I was finally able to create the LE cert

what I noticed is that the issue seems to be related to the hostname.

from the Debian CLI what is your output for the command 'hostname' ? does it look like mypbx.mydomain.com or simply mypbx

in my case it would fail each time if the sysadmin hostname was defined to use the full fqdn vs the hostname alone - FreePBX 16 had no issue defining the hostname field in sysadmin with the full fqdn however on Debian this seems to cause an issue for both LE cert creation and the vqplus module ( qcallback specifically, the system will go into a death loop )

nobe80 commented 5 months ago

@dolesec hostname => bitpbx

I get always the same result, doesnt matter what i do. Could this has something to do with this: Public IP: dns error ? Is there a logfile where i can see the lets encrypt error?

dolesec commented 5 months ago

interesting , I'm seeing the same result now when testing regardless of hostname definition

Public IP: dns error

ramarajan222 commented 5 months ago

Hi @dolesec @nobe80 ,

Let's Encrypt certificate generation issue has been resolved with the latest firewall module (17.0.1.17), Please upgrade the firewall module and give it a try again.

nobe80 commented 5 months ago

Hi @ramarajan222, unfortunately nothing has changed. Still the same mistake. Doenst matter if the hostname is

bitpbx or
bitpbx.beonit-cloud.de Also doenst matter if Firewall is on or off, still the same

kguptasangoma commented 5 months ago

Hi @nobe80

Can you please follow below steps and share the details -

1) Start the video of your UI so you can capture your steps.

2) Start the tcpdump on your system like tcpdump -i any -s0 -A port 80 -w /tmp/port80.pcap

3) follow steps to generate the LE cert via UI

4) Once you receive the error then stop the video and stop the packet capture.

5) share both the files along with output of your below commands - "iptables -nL lefilter" "fwconsole ma list"

Please note that this is publicly accessible platform so do not share any confidential information.

Regards Kapil

kguptasangoma commented 5 months ago

Hi @nobe80 any update on my above comment? Thanks

nobe80 commented 5 months ago

Hi @kguptasangoma

i installed today the latest updates but we have tzhe same results with Lets Encrypt.

So what do you mean? The best way would be i give you all the access credentials to our test server and you can see/change whatever you want?

kguptasangoma commented 5 months ago

Hi @nobe80

Can you please share the details as requested in https://github.com/FreePBX/issue-tracker/issues/115#issuecomment-2077349474 ?

nobe80 commented 5 months ago

HI @kguptasangoma

attached you have the video and the pcap file.

Video file: freepbx_le_certificate_error.mp4.zip

Pcap file: port80.pcap.zip

Point 5 is: root@bitpbx:~# iptables -nL lefilter iptables v1.8.9 (nf_tables): chain lefilter' in tablefilter' is incompatible, use 'nft' tool. root@bitpbx:~# fwconsole ma list No repos specified, using: [standard,unsupported] from last GUI settings

ramarajan222 commented 5 months ago

Hi @nobe80

Looks like the port 80 is not set to Letencrypt . Can you set port 80 to lets encrypt on sysadmin and then give it a try again?

nobe80 commented 5 months ago

Hi @kguptasangoma

that port is set to lets encrypt, pls see my screenshot.

dolesec commented 5 months ago

@nobe80 do you see any differences when running the following ...

iptables-save | grep lefilter vs iptables-nft-save | grep lefilter

nobe80 commented 5 months ago

Hi @dolesec

root@bitpbx:~# iptables-save | grep lefilter root@bitpbx:~# iptables-nft-save | grep lefilter root@bitpbx:~#

ramarajan222 commented 5 months ago

Hi @nobe80 ,

Curl request to port 80 is getting a timeout to your pbx. something's wrong, any idea why the curl request is getting timed out? curl -I http://bitpbx.beonit-cloud.de/ curl: (7) Failed to connect to bitpbx.beonit-cloud.de port 80: Connection timed out

See my working pbx returned 403 curl -I http://letest1.sangomaqa.com/ HTTP/1.1 403 Forbidden Date: Mon, 06 May 2024 13:57:58 GMT Server: Apache Content-Type: text/html; charset=iso-8859-1

nobe80 commented 5 months ago

Hi @ramarajan222

maybe that is another issue? When i use curl i get:

Normans-MacBook-Pro ~ % curl -I http://bitpbx.beonit-cloud.de HTTP/1.1 403 Forbidden Date: Mon, 06 May 2024 14:16:00 GMT Server: Apache Content-Type: text/html; charset=iso-8859-1

nobe80 commented 5 months ago

I disable fail2ban to be sure but i get the same result:

Processing: bitpbx.beonit-cloud.de, Local IP: 116.203.228.6, Public IP: dns error Self test: trying http://bitpbx.beonit-cloud.de/.freepbx-known/d0ab4b3b5cf0e6c664156d0ca2830a9a Self test: received d0ab4b3b5cf0e6c664156d0ca2830a9a

kguptasangoma commented 5 months ago

Hi @nobe80 for sure it looks like something is not right from your system environment point of view.

In a default debian system which we are using has ipset and iptable commands

ipset --list -> will show the port 80 configuration and also firewall rule in iptables-save command.

When certman starts the LE cert generation process then - 1) it will first open the port 80 ( can confirm by ipset --list command) for 1 min. 2) during this 1 min, mirror server sends the request to the host to validate the dns 3) now in your case its timing out because port 80 is somehow not reachable .

please check at your end if port 80 is open to listen LE messages or not.

nobe80 commented 5 months ago

Hi @kguptasangoma,

Natural Debian 12 Server: root@bitpbx:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm

Connect via Telnet form my Macbook to the freepbx in the cloud: norman@Normans-MacBook-Pro ~ % telnet bitpbx.beonit-cloud.de 80 Trying 116.203.228.6... Connected to bitpbx.beonit-cloud.de. Escape character is '^]'.

Netstat output root@bitpbx:~# netstat -tulpen Active Internet connections Proto Recv-Q Send-Q Local Address tcp 0 0 127.0.0.1:6006 tcp 0 0 127.0.0.1:6001 tcp 0 0 127.0.0.1:8088 tcp 0 0 127.0.0.1:6086 tcp 0 0 127.0.0.1:6081 tcp 0 0 0.0.0.0:1720 tcp 0 0 127.0.0.1:3306 tcp 0 0 0.0.0.0:6002 tcp 0 0 0.0.0.0:8080 tcp 0 0 0.0.0.0:25 tcp 0 0 0.0.0.0:84 tcp 0 0 0.0.0.0:82 tcp 0 0 0.0.0.0:83 tcp 0 0 0.0.0.0:80 tcp 0 0 0.0.0.0:81 tcp 0 0 127.0.0.1:5038 tcp 0 0 127.0.0.1:6379 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:4573 tcp 0 0 0.0.0.0:54222 tcp6 0 0 :::8003 tcp6 0 0 :::8001 tcp6 0 0 :::8089 tcp6 0 0 :::6082 tcp6 0 0 :::6083 tcp6 0 0 :::25 tcp6 0 0 :::54222 udp 0 0 0.0.0.0:48995 udp 0 0 0.0.0.0:68 udp 0 0 0.0.0.0:69 udp 0 0 127.0.0.1:323 udp 0 0 0.0.0.0:4520 udp 0 0 0.0.0.0:4569 udp 0 0 0.0.0.0:5000 udp 0 0 0.0.0.0:5060 udp6 0 0 :::69 udp6 0 0 ::1:323 udp6 0 0 :::35715 root@bitpbx:~# (only servers) Foreign Address State User Inode PID/Program name
0.0.0.0: LISTEN 999 99807 21315/node /var/www 0.0.0.0: LISTEN 999 99806 21315/node /var/www 0.0.0.0: LISTEN 999 20625 1782/asterisk
0.0.0.0: LISTEN 999 99934 21315/node /var/www 0.0.0.0: LISTEN 999 99931 21315/node /var/www 0.0.0.0: LISTEN 999 19810 1782/asterisk
0.0.0.0: LISTEN 104 15903 826/mariadbd
0.0.0.0: LISTEN 0 15685 745/apache2
0.0.0.0: LISTEN 0 15673 745/apache2
0.0.0.0: LISTEN 0 17795 1354/master
0.0.0.0: LISTEN 0 15681 745/apache2
0.0.0.0: LISTEN 0 15679 745/apache2
0.0.0.0: LISTEN 0 15677 745/apache2
0.0.0.0: LISTEN 0 15683 745/apache2
0.0.0.0: LISTEN 0 15675 745/apache2
0.0.0.0: LISTEN 999 20627 1782/asterisk
0.0.0.0: LISTEN 102 15845 706/redis-server 12 0.0.0.0: LISTEN 109 16960 693/mongod
0.0.0.0: LISTEN 999 21537 2548/node /var/www/ 0.0.0.0: LISTEN 0 15775 751/sshd: /usr/sbin ::: LISTEN 999 23383 2875/node /var/www/ ::: LISTEN 999 23382 2875/node /var/www/ ::: LISTEN 999 20626 1782/asterisk
::: LISTEN 999 99916 21315/node /var/www ::: LISTEN 999 99941 21315/node /var/www ::: LISTEN 0 17796 1354/master
::: LISTEN 0 15785 751/sshd: /usr/sbin 0.0.0.0: 999 19698 1782/asterisk
0.0.0.0: 0 14287 537/dhclient
0.0.0.0: 0 16826 770/in.tftpd
0.0.0.0: 0 15810 785/chronyd
0.0.0.0: 999 19817 1782/asterisk
0.0.0.0: 999 19806 1782/asterisk
0.0.0.0: 999 19855 1782/asterisk
0.0.0.0: 999 19693 1782/asterisk
::: 0 16827 770/in.tftpd
::: 0 15811 785/chronyd
::: 999 19699 1782/asterisk

So i can connect to port 80 from outside. So firewall and fail2ban is disabled. The Debian 12 is a fresh installed Debian 12 system. What else can I do?

kguptasangoma commented 5 months ago

Hi @nobe80 PBX is continuously replying 404 not found , see below snapshot from your shared pcap.

Could you please update the sysadmin to the latest edge and restart apache2 and try again just in case to rule out apache2 permission issue ?

nobe80 commented 5 months ago

Hi @kguptasangoma

could it something to do that the direcory is empty in /var/www/html/.well-known/acme-challenge ?

kguptasangoma commented 5 months ago

Not really sure.

Are you creating debian in any cloud vm or your own local system?

kguptasangoma commented 5 months ago

you could try to create debian on vultr and give a try there please.

nobe80 commented 5 months ago

So if i create a test html file in there then it works: http://bitpbx.beonit-cloud.de/.well-known/acme-challenge/test.html Can you confirm this?

vultr i have never use that here in germany...

dolesec commented 5 months ago

@nobe80 the file created for the LE process is created and removed in short order , it doesn't stay around ...

I was able to view your Test document

dolesec commented 5 months ago

Hi @dolesec

root@bitpbx:~# iptables-save | grep lefilter root@bitpbx:~# iptables-nft-save | grep lefilter root@bitpbx:~#

@nobe80 that's really odd , something is genuinely unique with your install; are you running version 16 successfully in a VM with that same provider where certman has no issues with LE ?

if you can I would try with a provider like vultr.com or linode.com

kguptasangoma commented 5 months ago

Hi @nobe80 I agree with @dolesec here, something odd in your environment, may be worth to give try with 16 our distro iso to see if that works fine or not. or else give a try with 17 on digital ocean or aws if vultr is not a viable option for you.

Thanks

nobe80 commented 5 months ago

Hi @kguptasangoma, @dolesec

we have here a natural Debian 12 without any firewall or fail2ban enabled. Only freepbx is installed there. There is a technical reason why LE doesnt work. Yes of course we have a lot of another VM´s that use LE certificate. One of the most example is Plesk with Lets encrypt. We got never problems with that since years. Using vultr is not an option due to gdpr reasons.

So is there any Debug logfile that we can see the creation or that sopecific url? Maybe it takes to short or something else? If you guys want i can give you the credentials ssh and freepbx admin.

I will try another certbot for lets encypt.

regards

dolesec commented 5 months ago

@nobe80 you can view the output by using the command line options from the CLI and turning up verbosity ...

with respect to testing using something like vultr ... you don't have to use it but you could test this process; it might shed some light on the differences impacting your installation

root@fpbx17:~# fwconsole cert
Description:
  Certificate Management

Usage:
  certificates [options]

Options:
      --list                       List Certificates
      --updateall                  Check and Update all Certificates
      --force                      Force update, by pass 30 days expiry 
      --import                     Import any unmanaged certificates in /etc/asterisk/keys
      --generate                   Generate Certificate
      --type=TYPE                  Certificate generation type - "le" for LetsEncrypt
      --hostname=HOSTNAME          Certificate hostname (LetsEncrypt Generation)
      --country-code=COUNTRY-CODE  Country Code (LetsEncrypt Generation)
      --state=STATE                State/Provence/Region (LetsEncrypt Generation)
      --email=EMAIL                Owner's email (LetsEncrypt Generation)
      --san=SAN                    Certificate Subject Alternative Name(s) (LetsEncrypt Generation) (multiple values allowed)
      --delete=DELETE              Delete certificate by id or hostname
      --default=DEFAULT            Set default certificate by id or hostname
      --details=DETAILS            Display certificate details by id or hostname
      --json                       Format output as json
  -h, --help                       Display help for the given command. When no command is given display help for the list command
  -q, --quiet                      Do not output any message
  -V, --version                    Display this application version
      --ansi|--no-ansi             Force (or disable --no-ansi) ANSI output
  -n, --no-interaction             Do not ask any interactive question
  -v|vv|vvv, --verbose             Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

nobe80 commented 4 months ago

Hi there,

in the meantime i tried certbot and have installed certbot on Debian 12 like this docu: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-20-04

After that it works! Lets encrypt works fine and now i have an encrpted URL: https://bitpbx.beonit-cloud.de

So that means it has something to do with freepbx, not with the server we have. Can we check that again?

regards

dolesec commented 4 months ago

@nobe80 if that works in your environment this post will serve others well that find themselves in a similar scenario...

in my continued testing with 3 different virtual platform providers I am not finding any issues with the current module - the LE cert creation/renewal process is working as expected ( Vulture, Linode , AWS )

it also looks like the site you referenced only handles apache; for many others the cert may also be used for SIP TLS - have you gone down that path with this other method as well ?

Regardless good job unraveling this, the key difference I see is your systems use of UFW which is not enabled by default according to my research.

nobe80 commented 4 months ago

@dolesec thanks but i dont think my problem has something to do with the virtual platfom. The platform that my datacenter use is KVM by the way.

No i don't yet secure SIP TLS with that Lets Encrapt certificate because i havent found a practical method to get it work. We would like to sell freebpx V17 soon. As far as i know i dont use UFW but i will double check that.

Question: is ip-tables needed for lets encrypt?! we dont need the firewall, only fail2ban

FreePBX / issue-tracker