SSO stops working after a while on Fortigate

lyubomirtraykov commented 1 year ago

Describe the bug In Packet Fence 12 after a while the sso daemon stops sending accounting request to the Fortigate. The only anomaly that I see in the pfsso.log is:

Oct 31 11:56:17 pf-01 pfsso-docker-wrapper[2935]: t=2022-10-31T11:56:17+0200 lvl=info msg="Calling Unified API on uri: http://100.64.0.1:22226/api/v1/pfconnector/dynreverse" pid=7 request-uuid=43aa3272-5902-11ed-9c86-024264400007 username=USER ip=10.10.100.56 mac=XX:XX:XX:XX:XX:XX role=Employee firewall-id=10.10.10.1
Oct 31 11:56:17 pf-01 pfsso-docker-wrapper[2935]: t=2022-10-31T11:56:17+0200 lvl=info msg="Reusing existing port 45707" pid=7 request-uuid=43aa3272-5902-11ed-9c86-024264400007 username=USER ip=10.10.100.56 mac=XX:XX:XX:XX:XX:XX role=Employee firewall-id=10.10.10.1
Oct 31 11:56:22 pf-01 pfsso-docker-wrapper[2935]: t=2022-10-31T11:56:22+0200 lvl=eror msg="Couldn't SSO to the fortigate, got the following error: context deadline exceeded" pid=7 request-uuid=43aa3272-5902-11ed-9c86-024264400007 username=USER ip=10.10.100.56 mac=XX:XX:XX:XX:XX:XX role=Employee firewall-id=10.10.10.1
Oct 31 11:56:22 pf-01 pfsso-docker-wrapper[2935]: t=2022-10-31T11:56:22+0200 lvl=eror msg="Error while sending SSO to context deadline exceeded: %!s(MISSING)10.10.10.1" pid=7 request-uuid=43aa3272-5902-11ed-9c86-024264400007 username=USER ip=10.10.100.56 mac=XX:XX:XX:XX:XX:XX role=Employee

To Reproduce Steps to reproduce the behavior:

Configure Fortigate SSO
See errors in pfsso.log after a while

Expected behavior The SSO to work without restarting the service.

julsemaan commented 1 year ago

Please provide your package version

lyubomirtraykov commented 1 year ago

I have the latest pakages:


MariaDB-client.x86_64                         10.5.15-1.el8                                             @packetfence                     
MariaDB-common.x86_64                         10.5.15-1.el8                                             @packetfence                     
MariaDB-server.x86_64                         10.5.15-1.el8                                             @packetfence                     
MariaDB-shared.x86_64                         10.5.15-1.el8                                             @packetfence                     
MySQL-python.x86_64                           2.0.3-12.1                                                @packetfence                     
berkeleydb-ltb.x86_64                         4.6.21.NC-6.2                                             @packetfence                     
containerd.io.x86_64                          1.5.11-3.1.el8                                            @packetfence                     
dkms.noarch                                   2.8.4-1.el8                                               @packetfence                     
dkms-ipt-netflow.noarch                       2.6-13.1                                                  @packetfence                     
docker-ce.x86_64                              3:20.10.14-3.el8                                          @packetfence                     
docker-ce-cli.x86_64                          1:20.10.14-3.el8                                          @packetfence                     
docker-ce-rootless-extras.x86_64              20.10.14-3.el8                                            @packetfence                     
docker-scan-plugin.x86_64                     0.17.0-3.el8                                              @packetfence                     
fingerbank.noarch                             4.3.2-20220823160915.620786098.0008.v4~3~2.el8            @packetfence                     
fingerbank-collector.x86_64                   1.4.0-1.el8                                               @packetfence                     
fping.x86_64                                  5.0-8.2                                                   @packetfence                     
freeradius.x86_64                             3.2.1-1.el8                                               @packetfence                     
freeradius-config.x86_64                      3.2.1-1.el8                                               @packetfence                     
freeradius-ldap.x86_64                        3.2.1-1.el8                                               @packetfence                     
freeradius-mysql.x86_64                       3.2.1-1.el8                                               @packetfence                     
freeradius-perl.x86_64                        3.2.1-1.el8                                               @packetfence                     
freeradius-redis.x86_64                       3.2.1-1.el8                                               @packetfence                     
freeradius-rest.x86_64                        3.2.1-1.el8                                               @packetfence                     
freeradius-utils.x86_64                       3.2.1-1.el8                                               @packetfence                     
galera-4.x86_64                               26.4.11-1.el8                                             @packetfence                     
haproxy.x86_64                                2.2.14-1.1                                                @packetfence                     
hiredis.x86_64                                0.13.3-16.2                                               @packetfence                     
ipt-netflow.x86_64                            2.6-13.1                                                  @packetfence                     
lasso.x86_64                                  2.7.0-8.2                                                 @packetfence                     
libapreq2.x86_64                              2.16-1.el8                                                @packetfence                     
libapreq2-libs.x86_64                         2.16-1.el8                                                @packetfence                     
libcollectdclient.x86_64                      5.9.0-5.4                                                 @packetfence                     
libssh2.x86_64                                1.9.0-5.el8                                               @packetfence                     
libtomcrypt.x86_64                            1.18.2-5.el8                                              @packetfence                     
libtommath.x86_64                             1.1.0-1.el8                                               @packetfence                     
mod_perl.x86_64                               2.0.11-1.el8                                              @packetfence                     
monit.x86_64                                  5.26.0-1.el8                                              @packetfence                     
netdata.x86_64                                1.10.0-70.2                                               @packetfence                     
openldap-ltb.x86_64                           2.4.45-5.1                                                @packetfence                     
openvas-cli.x86_64                            1.4.4-1.2                                                 @packetfence                     
openvas-libraries.x86_64                      8.0.8-4.2                                                 @packetfence                     
packetfence.x86_64                            12.0.0-20221024203127.675981810.0008.maintenance~12~0.el8 @packetfence                     
packetfence-perl.x86_64                       1.2.1-1.el8                                               @packetfence                     
packetfence-release.noarch                    2.4.1-20221024203127.675981810.0008.maintenance~12~0.el8  @packetfence                     
packetfence-upgrade.noarch                    12.0.0-20221024203127.675981810.0008.maintenance~12~0.el8 @packetfence                     
perl-BerkeleyDB.x86_64                        0.63-3.6                                                  @packetfence                     
perl-Cache-BDB.noarch                         0.04-5.1                                                  @packetfence                     
perl-Carp.noarch                              1.42-396.el8                                              @packetfence                     
perl-Convert-BinHex.noarch                    1.125-13.el8                                              @packetfence                     
perl-Crypt-OpenSSL-PKCS12.x86_64              1:1.7-5.1                                                 @packetfence                     
perl-Crypt-SMIME.x86_64                       0.27-12.1                                                 @packetfence                     
perl-Data-Dumper.x86_64                       2.167-399.el8                                             @packetfence                     
perl-Data-MessagePack-Stream.x86_64           0.07-4.1                                                  @packetfence                     
perl-Data-OptList.noarch                      0.110-6.el8                                               @packetfence                     
perl-Digest.noarch                            1.17-395.el8                                              @packetfence                     
perl-Digest-MD5.x86_64                        2.55-396.el8                                              @packetfence                     
perl-Encode.x86_64                            4:2.97-3.el8                                              @packetfence                     
perl-Exporter.noarch                          5.72-396.el8                                              @packetfence                     
perl-File-Path.noarch                         2.15-2.el8                                                @packetfence                     
perl-File-Temp.noarch                         0.230.600-1.el8                                           @packetfence                     
perl-Getopt-Long.noarch                       1:2.50-4.el8                                              @packetfence                     
perl-HTTP-Tiny.noarch                         0.074-1.el8                                               @packetfence                     
perl-IO-SessionData.noarch                    1.03-16.el8                                               @packetfence                     
perl-IO-Socket-IP.noarch                      0.39-5.el8                                                @packetfence                     
perl-MIME-Base64.x86_64                       3.15-396.el8                                              @packetfence                     
perl-MIME-tools.noarch                        5.509-9.el8                                               @packetfence                     
perl-Net-ARP.x86_64                           1.0.6-1.1                                                 @packetfence                     
perl-Net-DHCP.noarch                          0.696-23.1                                                @packetfence                     
perl-Net-Interface.x86_64                     1.012-8.1                                                 @packetfence                     
perl-Net-Nessus-REST.noarch                   0.7.0-3.1                                                 @packetfence                     
perl-Net-Nessus-XMLRPC.noarch                 0.40-7.1                                                  @packetfence                     
perl-Net-Pcap.x86_64                          0.18-21.5                                                 @packetfence                     
perl-NetPacket.noarch                         1.7.2-8.6                                                 @packetfence                     
perl-POSIX-2008.x86_64                        0.03-3.1                                                  @packetfence                     
perl-POSIX-AtFork.x86_64                      0.02-6.2                                                  @packetfence                     
perl-PathTools.x86_64                         3.74-1.el8                                                @packetfence                     
perl-Pod-Escapes.noarch                       1:1.07-395.el8                                            @packetfence                     
perl-Pod-Perldoc.noarch                       3.28-396.el8                                              @packetfence                     
perl-Pod-Simple.noarch                        1:3.35-395.el8                                            @packetfence                     
perl-Pod-Usage.noarch                         4:1.69-395.el8                                            @packetfence                     
perl-SOAP-Lite.noarch                         1.27-7.el8                                                @packetfence                     
perl-Scalar-List-Utils.x86_64                 3:1.53-440.6                                              @packetfence                     
perl-Socket.x86_64                            4:2.027-3.el8                                             @packetfence                     
perl-Storable.x86_64                          1:3.11-3.el8                                              @packetfence                     
perl-Term-ANSIColor.noarch                    4.06-396.el8                                              @packetfence                     
perl-Term-Cap.noarch                          1.17-395.el8                                              @packetfence                     
perl-Test-Simple.noarch                       1:1.302188-1.1                                            @packetfence                     
perl-Text-ParseWords.noarch                   3.30-395.el8                                              @packetfence                     
perl-Text-Tabs+Wrap.noarch                    2013.0523-395.el8                                         @packetfence                     
perl-Time-Local.noarch                        1:1.280-1.el8                                             @packetfence                     
perl-URI.noarch                               1.73-3.el8                                                @packetfence                     
perl-Unicode-Normalize.x86_64                 1.25-396.el8                                              @packetfence                     
perl-WWW-Curl.x86_64                          4.17-25.5                                                 @packetfence                     
perl-constant.noarch                          1.33-396.el8                                              @packetfence                     
perl-lasso.x86_64                             2.7.0-8.2                                                 @packetfence                     
perl-libapreq2.x86_64                         2.16-1.el8                                                @packetfence                     
perl-libnet.noarch                            3.11-3.el8                                                @packetfence                     
perl-parent.noarch                            1:0.237-1.el8                                             @packetfence                     
perl-podlators.noarch                         4.11-1.el8                                                @packetfence                     
perl-threads.x86_64                           1:2.21-2.el8                                              @packetfence                     
perl-threads-shared.x86_64                    1.58-2.el8                                                @packetfence                     
python3-impacket.noarch                       0.9.22-4.el8                                              @packetfence                     
python3-ldap3.noarch                          2.8.1-2.el8                                               @packetfence                     
python3-pycryptodomex.x86_64                  3.9.7-1.el8                                               @packetfence                     
sscep.x86_64                                  0.9.0-10.2                                                @packetfence                     
tcp_wrappers-libs.x86_64                      7.6-96.el8                                                @packetfence

julsemaan commented 1 year ago

More info will be needed to troubleshoot then.

When the issue happens, can you provide:

A full pcap of the port in the line 'Reusing existing port' while some SSO is being sent using tcpdump -nlp -i any port XYZ -w problem.pcap
A full pcap from the server to your firewall host: tcpdump -nlp -i any host 10.10.10.1
The output of netstat -anlp | grep XYZ

That will help determine the problem

lyubomirtraykov commented 1 year ago

I think it is a bug with the fortigate. It returns Accounting response without Accounting OK attribute.

It is strange that it works for couple of days and then it suddenly stops.

lyubomirtraykov commented 1 year ago

This is the result of tcpdump -nlp -i any port XYZ -w problem.pcap problem.pcap.zip

This is the result of netstat -anlp | grep XYZ udp6 0 0 :::33759 :::* 3155/pfconnector

julsemaan commented 1 year ago

It does indeed look like the request/response does work.

Are you able to see any SSO failures on the Fortigate due to this?

lyubomirtraykov commented 1 year ago

No it is working fine for couple of days and then it suddenly stops. And if I restart the server it starts working again.

julsemaan commented 1 year ago

It doesn't look like it has the right destination IP in the packets so it's not sending it to the local connector. I'll deep dive a bit more to see what I can find

lyubomirtraykov commented 1 year ago

This morning it happened again. But this time I was able to find different errors in log:

Nov  7 09:59:10 pf-01 pfsso-docker-wrapper[2935]: t=2022-11-07T09:59:10+0200 lvl=info msg="Processing SSO Start" pid=7 request-uuid=0fc9dbed-5e72-11ed-9c86-024264400007 username=FFFFFFF ip=10.10.100.163 mac=80:e8:2c:XX:XX:XX role=Employee firewall-id=10.10.10.1
Nov  7 09:59:10 pf-01 pfsso-docker-wrapper[2935]: t=2022-11-07T09:59:10+0200 lvl=info msg="Calling Unified API on uri: http://100.64.0.1:22226/api/v1/pfconnector/dynreverse" pid=7 request-uuid=0fc9dbed-5e72-11ed-9c86-024264400007 username=FFFFFFF ip=10.10.100.163 mac=80:e8:2c:XX:XX:XX role=Employee firewall-id=10.10.10.1
Nov  7 09:59:10 pf-01 pfsso-docker-wrapper[2935]: t=2022-11-07T09:59:10+0200 lvl=info msg="Reusing existing port 36005" pid=7 request-uuid=0fc9dbed-5e72-11ed-9c86-024264400007 username=FFFFFFF ip=10.10.100.163 mac=80:e8:2c:XX:XX:XX role=Employee firewall-id=10.10.10.1
Nov  7 09:59:10 pf-01 pfsso-docker-wrapper[2935]: t=2022-11-07T09:59:10+0200 lvl=eror msg="Couldn't SSO to the fortigate, got the following error: read udp 100.64.0.7:46261->10.10.10.4:36005: read: connection refused" pid=7 request-uuid=0fc9dbed-5e72-11ed-9c86-024264400007 username=filip.vodenicharov ip=10.10.100.163 mac=80:e8:2c:XX:XX:XX role=Employee firewall-id=10.10.10.1
Nov  7 09:59:10 pf-01 pfsso-docker-wrapper[2935]: t=2022-11-07T09:59:10+0200 lvl=eror msg="Error while sending SSO to read udp 100.64.0.7:46261->10.10.10.4:36005: read: connection refused: %!s(MISSING)10.10.10.1" pid=7 request-uuid=0fc9dbed-5e72-11ed-9c86-024264400007 username=FFFFFFFFF ip=10.10.100.163 mac=80:e8:2c:XX:XX:XX role=Employee

julsemaan commented 1 year ago

According to what I see in the pcap and in the code, it may be this behavior here: https://github.com/inverse-inc/packetfence/issues/7239

Not sure why it would appear suddenly and not consistently though.

Basically, the wrong source IP is used for the answer back to pfsso due to how internal traffic is being routed between pfsso and pfconnector-server.

The packet does make it correctly from what I can tell, the answer is not seen by pfsso, thus the error

Can you confirm the SSO process is indeed still working even when this behavior is happening? That would confirm my theory 100%

Thanks,

julsemaan commented 1 year ago

Notes for myself if this is confirmed:

We need to see if we can adjust the iptables rules to make sure the reply uses the right IP address
Otherwise, we can add a query parameter that makes it so we ignore the PFCONNECTOR_SERVER_DYN_REVERSE_HOST variable in pfconnector-server for some requests - we need to make sure it's compatible with k8s deployments

lyubomirtraykov commented 1 year ago

Yes I can confirm that the SSO is working even when the error is logged.

julsemaan commented 1 year ago

Had a conversation with @fdurand, iptables can't do anything about it. It's the selection of the docker0 interface that causes this.

I'll look into implementing the conditional selection of 100.64.0.1 based on whether or not the container is local to the pfconnector-server

fdurand commented 1 year ago

I have made a change in iptables to change the destination ip if the packet is coming from docker and the destination is the management ip address of the serser. I can see on my test setup that the source ip/port and destination ip/port is ok and i don't have any icmp port unreachable after that (pfsso is happy). Are you able to test it and let me know if it works for you ?

https://github.com/inverse-inc/packetfence/compare/fix/7323

nqb commented 1 year ago

I want to mention that I see a similar behavior (RADIUS request rejected due to timeout) when testing CLI login on PacketFence using a RADIUS source configured to use local connector:

Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] handling radius autz request: from switch_ip => (192.168.0.1), connection_type => Ethernet-NoEAP,switch_mac => (Unknown), mac => [0], port => , us
ername => "bob" (pf::radius::switch_access)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) WARN: [mac:[undef]] Trying to match IP address with an invalid MAC address 'undef' (pf::ip4log::mac2ip)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Instantiate profile default (pf::Connection::ProfileFactory::_from_profile)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Found authentication source(s) : 'local,file1,cli_login_radius_source' for realm 'null' (pf::config::util::filter_authentication_sources)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] MFA Pre Authentication (pf::radius::mfa_pre_auth)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Instantiate profile default (pf::Connection::ProfileFactory::_from_profile)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Found authentication source(s) : 'local,file1,cli_login_radius_source' for realm 'null' (pf::config::util::filter_authentication_sources)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Using sources local, file1, cli_login_radius_source for matching (pf::authentication::match2)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Matched rule (catchall) in source cli_login_radius_source, returning actions. (pf::Authentication::Source::match_rule)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] Matched rule (catchall) in source cli_login_radius_source, returning actions. (pf::Authentication::Source::match)
Nov 16 01:27:40 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) ERROR: [mac:[undef]] unable to read password file '/usr/local/pf/conf/admin.conf' (pf::Authentication::Source::HtpasswdSource::authenticate)
Nov 16 01:27:42 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) ERROR: [mac:[undef]] Unable to perform RADIUS authentication on any server: ETIMEOUT (pf::Authentication::Source::RADIUSSource::_handle_radius_request)
Nov 16 01:27:42 pfel8dev httpd.aaa-docker-wrapper[73031]: httpd.aaa(6) INFO: [mac:[undef]] User bob tried to login in 192.168.0.1 but authentication failed (pf::radius::authenticate)

Not sure if it's the exact same issue. Will try fix/7323 to see if it helps.

julsemaan commented 1 year ago

@nqb, this can happen for this specific bug. Basically, all UDP based protocols are prone to break at some point due to that bug so that includes httpd.aaa talking to an external RADIUS server like in your example

nqb commented 1 year ago

I'm not able to replicate issue easily so it's hard to say that fix/7323 fix it but at least I don't see any issue.

julsemaan commented 1 year ago

I found an actual workaround that was put in the Perl code but not the Go code: https://github.com/inverse-inc/packetfence/blob/devel/lib/pf/connector.pm#L58-L63

Technically, applying this same logic to the Golang code should make it work even without iptables but will limit the usage of pfconnectors for SSO to standalone servers (already the case for auth sources)

What kind of confuses me is that some customers are reporting similar issues with httpd.aaa + SNMP but that looks to be another issue

inverse-inc / packetfence

SSO stops working after a while on Fortigate #7323