search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.13k stars 710 forks source link

Wildcard Support in Firewall Alias - Type Host(s) #4145

Closed Mer-idium closed 3 years ago

Mer-idium commented 4 years ago

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

[-] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[-] I have searched the existing issues and I'm convinced that mine is new.

Hi All,

Is it possible to add the feature to include wildcard support in Firewall Alias groups? At the moment rather than going *.somedomain.com to white list all subdomains i need to manually add each one like so: first.somedomain.com, second.somedomain.com etc...

This can get irritating when i have many subdomains to add which are near identical. In some instances the only way to obtain all domains is to run through Unbound logs to find the single sub domain i missed which drags out the process.

Thanks

AdSchellevis commented 4 years ago

long story short, you can't (easily) request all existing records for a (top) level domain. https://stackoverflow.com/questions/19322962/how-can-i-list-all-dns-records

Our aliases resolve on intervals and capture the associated addresses in a table, which only works if you can determine addresses up front.

You can add domain overrides in unbound and prevent the clients from resolving, which is what most of the blacklist solutions use as well. A more complex solution would be to capture unbounds responses and add those to a table, this is however outside the scope of our project.

Mer-idium commented 4 years ago

If the issue was for blacklisting i wouldn't mind as i would use unbound. This is more for traffic direction to negate a proxy and then pass the traffic through a VPN.

agh1467 commented 3 years ago

This is becoming a more common feature in firewalls. Several vendors already have support for it. As cloud usage expands, DNS names are becoming more dynamic by nature.

Here are several examples of the feature from other vendors: https://www.sonicwall.com/support/knowledge-base/dns-resolution-of-wildcard-fqdn-address-objects/170505295458240/ https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/policies/fqdn_about_c.html https://docs.fortinet.com/document/fortigate/6.2.0/cookbook/217973/using-wildcard-fqdn-addresses-in-firewall-policies

Trying to request all DNS records of a domain is a futile effort, as is trying to perform reverse DNS lookups. Generally, the way this feature works in other products is by listening to DNS requests as they pass through the device. The listening service keeps a local table of DNS names and IPs as appropriate for the aliases that have DNS names configured. Either the firewall consults the table when applying rules, or the DNS service updates the alias definitions as DNS requests come through. There are probably other implementations that would work as well.

The obvious challenge in wildcard-based rules is keeping everything in sync. A query now, may net different results from a query later. Queries performed by the firewall my net different replies from a client making the same DNS query. The firewall rules must always contain the exact IPs that are sent to DNS clients. A user in the pfsense reddit explained it fairly concisely.

https://www.reddit.com/r/PFSENSE/comments/7wwnun/wildcard_domains_in_aliases/du4t5t0/?utm_source=reddit&utm_medium=web2x&context=3

The way hostnames in aliases work is by periodically resolving those exact names every few minutes and keeping a pf table up-to-date. That's easy for a fully qualified domain name but impossible for wildcards. You can't query a DNS server and ask for *.example.com. This mechanism also fails for hostnames which have randomized record set replies, which is common with content hosted on CDNs.

It does not directly match the name when queried because that is not possible at the firewall level. It does not attempt to perform a reverse DNS lookup because the vast majority of the time that would be useless since reverse DNS would fail to match what the user expects in shared hosting space. You have to intercept the initial client DNS request and base a response on what the client received. pfBlockerNG and its DNSBL feature may help there, depending on what you're trying to do.

A common usage of this feature is to make dynamic rules to allow CDN based services. The domain names used by CDNs often change, come and go as demands ebb and flow. Using known FQDNs is just not an option as cloud DNS names are often ephemeral. This is different from the utility of a DNS blacklist because the decision to allow or deny is not made at the DNS resolver, but instead in a firewall rule designed to allow the client's traffic to reach its destination. Instead of preventing a client from reaching the destination, it's allowing a client to reach ONLY specific destinations.

Another usage of this feature is with policy-based routing, and sending traffic destined for Amazon Prime servers out my WAN interface instead of the VPN interface. Amazon Prime disables their service while connecting via a known VPN provider.

Here is an example of an Amazon Prime DNS name: avlrc-a1kaxig6vxsg8y-11d2d2020081910dandroid-nvidiadarcydarcy9p.na.api.amazonvideo.com

I wouldn't be surprised if this DNS name was created just for me, just to stream this one video, or even a segment of this one video. This could easily be captured in a wildcard entry such as *.amazonvideo.com

AdSchellevis commented 3 years ago

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.

s3rvant commented 3 years ago

Agreed with @agh1467 that wildcard support would be great

Our primary firewall is proprietary and supports wildcard domains however I'm unable to completely move to pfSense because that support is required by some of our services

marcelmah commented 3 years ago

I agree, would be nice, in my use case I would need it to create a firewall rule to allow *.api.letsencrypt.com to connect to my NGINX to validate LetsEncrypt certs. My OPNsense only allows one country (GeoIP) to connect to my webserver by default. Now I need to enable two rules to allow everyone to connect to validate and then I can disable them again. It's not much, but it's manual labor...

gongoscho commented 2 years ago

Is there already a feature request for this?

Will only allow the company cell phone to connect to port 443 * .eu.blackberry.net via TCP and nothing more.

Unfortunately I have to enter the server IPs manually.

mimugmail commented 2 years ago

This only works for proxy as the firewall cant guess any FQDN

gongoscho commented 2 years ago

That is not right 🙃 Take a look at the documentation 🤓

I created an alias: BlackBerry_Servers, and added the addresses:

which are also resolved:

under: Firewall: Diagnostics: Aliases 😎 and the alias work in the firewall-rules

mimugmail commented 2 years ago

But you cannot create one alias *.Blackberry.net which would resolve all possible FQDNs into one list ;)

derritter88 commented 2 years ago

I am currently evaluating OPNsense as I might migrate from Sophos XG to it. At Sophos XG this feature is working without an issue and I really would love to see it also working at OPNsense.

derritter88 commented 2 years ago

Maybe this tool might be a good starting point? -> Knock https://github.com/guelfoweb/knock

or

https://securitytrails.com/blog/subdomain-scanner-find-subdomains

derritter88 commented 2 years ago

I just used Knock and all the hits a outputted are the same as on Sophos XG.

So maybe a solution?!

Althought I don't know if BSD can run Python3

inferKNOX commented 2 years ago

As @agh1467 mentioned, it works flawlessly in the Fortigate I use.

Basically, once the wildcard is set in the subdomain, the firewall seems to wait for DNS queries of any subdomains that match the wildcard, and once it gets one + a hit on the firewall rule, it resolves it and caches the sub-domain and IP. My impression is that it is either, that or perhaps it does a reverse lookup of new IPs before they are processed by the rules, then if the resolved IP matches the wildcard, it caches that specific IP against the wildcard & proceeds to process firewall rules. It then seemingly periodically queries the cached subdomain and updates the IP if necessary, or goes back to an unresolved state if resolution fails for a few tries.

Whatever the case, wildcard FQDNs remain unresolved until there is an attempt to access a matching subdomain and the wildcard FQDN receives a hit by the firewall rules. This is invaluable in policy-based routing, particularly for CDNs, and is a major roadblock (for me) in migrating to OPNsense.

mimugmail commented 2 years ago

Wow .. what would happen if you brute force a domain with a wildcard A record 😀

inferKNOX commented 2 years ago

Do you mean that such a function introduces a vulnerability, and firewalls supporting it are exposed to that risk?

AdSchellevis commented 2 years ago

This just only works if you either snif dns traffic to feed the alias logic or let the resolver (unbound) pass the requested domains. Since we fetch in advance, you can't really do wildcards like this... but I think I already explained that. when it's not really a fine grained policy restriction (lock youtube for all), an alias in the resolver usually does the trick as well.

Sniffing / listing to dns requests (which is probably what the others do) will eventually stop working as well when more and more dns traffic is being encrypted anyway.

modest commented 1 year ago

@inferKNOX You can accomplish this today using dnsmasq.

  1. Set dnsmasq as your opnsense DNS server. (Or, if you really want to continue using Unbound, you can set Query Forwarding in Unbound to forward lookups for specific wildcard domains to dnsmasq internally.)
  2. In Firewall > Aliases, create an empty alias such as hosts_from_dns. Set the type to External (Advanced) with both IPv4 and IPv6.
  3. Add a custom config file for dnsmasq in /usr/local/etc/dnsmasq.conf.d/dnsmasq-ipset.conf 
    
# Add the response for certain A/AAAA lookups to an opnsense alias
ipset=/bbc.com/bbc.co.uk/bbci.co.uk/hosts_from_dns

Uncomment these if Unbound is still your primary DNS server; otherwise you'll have a loop

no-resolv

server=8.8.8.8


4. Restart dnsmasq.

Run a test DNS query, then check the contents of the new table (Firewall > Diagnostics > Aliases > hosts_from_dns) to see it populate.  You can use this alias (hosts_from_dns) in firewall rules.

(Despite the config option being named "ipset", dnsmasq will use BSD pf tables or nftables depending on the OS.)

_Edit: Updated steps to clarify that the type should be "External (Advanced)" and commented out the Unbound loop mitigation by default. Removed the mitigation for the Aliases Resolver Interval._
AdSchellevis commented 1 year ago

In Firewall > Aliases, create an empty alias such as hosts_from_dns

Make sure you choose External (advanced) as type in this case as these won't be touched by the system. Nice tip by the way.

agh1467 commented 1 year ago

Hey all, I tried @modest 's solution using dnsmasq and it does function and supports subdomains.

dnsmasq log sample creating entries for the primary domain, and subdomains equally:

ipset add hosts_from_dns 108.157.4.90 amazonvideo.com 
info: table created 
1 addresses added 
reply amazonvideo.com is 108.157.4.90 
ipset add hosts_from_dns 108.157.4.59 amazonvideo.com 
1 addresses added 
reply amazonvideo.com is 108.157.4.59 
ipset add hosts_from_dns 108.157.4.77 amazonvideo.com 
1 addresses added 
reply amazonvideo.com is 108.157.4.77 
ipset add hosts_from_dns 108.157.4.60 amazonvideo.com 
1 addresses added 
reply amazonvideo.com is 108.157.4.60 
reply amazonvideo.com is NODATA-IPv6 
reply avlrc-a1kaxig6vxsg8y-11d2d2020081910dandroid-nvidiadarcydarcy9p.na.api.amazonvideo.com is <CNAME> 
reply atv-ext.amazon.com is <CNAME> 
reply v1qxr2rxh3.na.prod.ter.amazonvideo.com is <CNAME> 
reply na.t100.prod.ter.amazonvideo.com is <CNAME> 
ipset add hosts_from_dns 34.231.95.236 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 34.231.95.236 
ipset add hosts_from_dns 18.209.234.142 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 18.209.234.142 
ipset add hosts_from_dns 34.226.163.160 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 34.226.163.160 
ipset add hosts_from_dns 3.228.121.198 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 3.228.121.198 
ipset add hosts_from_dns 3.95.140.135 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 3.95.140.135 
ipset add hosts_from_dns 3.86.124.162 iad.t100.prod.ter.amazonvideo.com 
1 addresses added 
reply iad.t100.prod.ter.amazonvideo.com is 3.86.124.162 
reply iad.t100.prod.ter.amazonvideo.com is NODATA-IPv6

All of the IPs are grouped together in the table.

pfctl -t hosts_from_dns -T show:

   3.86.124.162
   3.95.140.135
   3.228.121.198
   18.209.234.142
   34.226.163.160
   34.231.95.236
   108.157.4.59
   108.157.4.60
   108.157.4.77
   108.157.4.90

It does work for both NAT/Outbound, and Rules.

In Firewall > Advanced, set Aliases Resolve Interval to 0 to disable it. There is a bug where it clears unrecognized tables.

I tried this set up with and without this and functionality was the same, and I did not experience the table being cleared.

An apparent limitation of this dnsmasq functionality is that the TTL for DNS records is not considered, and IPs added to the set remain in the set forever. Used with a CDN service, the table may become huge. I'd guess it may have a negative impact on performance at some point depending on hardware, the number of records in the table, and the frequency and number of requests made by clients.

The only configuration file modification that was necessary for this was to specify a single ipset value: ipset=/amazonvideo.com/hosts_from_dns

For reference:

       --ipset=/<domain>[/<domain>...]/<ipset>[,<ipset>...]
              Places the resolved IP addresses of queries for one or more
              domains in the specified Netfilter IP set. If multiple setnames
              are given, then the addresses are placed in each of them,
              subject to the limitations of an IP set (IPv4 addresses cannot
              be stored in an IPv6 IP set and vice versa).  Domains and
              subdomains are matched in the same way as --address.  These IP
              sets must already exist. See ipset(8) for more details.

Other specific configurations, were available in the web UI. However, there is no interface for specifying ipset values. I'd guess that wouldn't happen until the interface is updated to MVC, where it could use an ArrayField with a CSVListField for storing the domain names.

I saw in the logs that dnsmasq created the table when it didn't exist, which is good, and it would just add IPs to a given table that already exists, so it doesn't seem like there would be a risk of clobbering an existing table.

And as @AdSchellevis mentioned, the External type should be used here. Since the pf tables populated by dnsmasq/ipset and the aliases look the same, I'd guess that they would function similarly (only the content wouldn't be stored in the config, and would clear periodically whenever that alias content is saved), I'd also guess that's why Aliases Resolve Interval is being set to 0 to prevent that process from clearing the table.

All in all, it seems like a pretty good work around with some limitations which should be considered on a per-implementation basis.

I wonder if this functionality could be integrated into the Aliases interface...hmmm maybe using a new Alias type?

modest commented 1 year ago

Make sure you choose External (advanced) as type in this case as these won't be touched by the system. Nice tip by the way.

Thanks. I've confirmed that the Aliases Resolve Interval no longer clears the table/alias when the type is set to External. Updated my comment to remove this mitigation.

An apparent limitation of this dnsmasq functionality is that the TTL for DNS records is not considered, and IPs added to the set remain in the set forever. Used with a CDN service, the table may become huge. I'd guess it may have a negative impact on performance at some point depending on hardware, the number of records in the table, and the frequency and number of requests made by clients.

This is true. And, just to get ahead of anyone who might try the an incorrect solution: While pfctl can remove old entries (-T expire 86400 for over 24 hours old), these are not safe to remove. These "old" entries may have been recently served by dnsmasq even seconds ago. After dnsmasq ipset adds an IP address, the table timestamp will not be updated again on subsequent queries.

There's a (very ugly) workaround to the infinite growth issue using 2 tables/aliases. You can either (a) have dnsmasq add all IPs to 2 ipsets (e.g. hosts_from_dns_a and hosts_from_dns_b - this can be done on one line with a comma), with a cron script that flushes one of them, alternating, every day or (b) write a cron script that clones dnsmasq's ipset table (e.g. hosts_from_dns and hosts_from_dns_older) before flushing dnsmasq's ipset table. Firewall rules would need to use both tables.

However, there is no interface for specifying ipset values.

Please please please :)

Sniffing / listing to dns requests (which is probably what the others do) will eventually stop working as well when more and more dns traffic is being encrypted anyway.

Tip: You can tell dnsmasq to add all IP addresses to an ipset with ipset=//all_dns_hosts. You can use this in combination with a firewall log rule (destination NOT all_dns_hosts) to catch any cases where a device/app is bypassing your DNS server. Those are either using their own DNS servers, eDNS servers, or hardcoded IP addresses.

mlazzarotto commented 1 year ago

Hello, I wish to introduce this functionality within my OPNsense.
Can I make multiple aliases for different servers?
For GNS3, I may want to allow pypi.org and github.com; for Docker containers I may want to allow hub.docker.com, and so on.