Data exfiltration via DNS tunneling

[TonyWildish-BH]

Describe the bug We have a guy doing penetration testing for us on our TRE, and he’s discovered that the DNS configuration is vulnerable, allowing DNS tunneling. He’s successfully exfiltrated data with iodine.

We tried enabling Defender for DNS, which it turns out doesn’t exist anymore, so I enabled Microsoft Defender for Servers Plan 2 instead, which we thought might give us the same protection. It doesn’t, he was still able to exfiltrate data.

Is there any MS service we can enable that will prevent DNS tunneling from the TRE?

Steps to reproduce

Attacker-side:

1. Register a domain and create a VM on the Internet
2. Setup an NS DNS record to point towards the VM
3. Deploy Iodine server (10.0.0.1)

On the Workspace VM (only tested Linux at the moment)

1. Download iodine client via apt on Ubuntu
2. Establish a DNS tunnel using Iodine to the server
3. An SFTP session was then able to exfiltrate 100 MB in about 2 and a half hours.

Azure TRE release version (e.g. v0.14.0 or main): main, as of about May.

Deployed Azure TRE components - click the (i) in the UI: UI Version: 0.5.21 API Version: 0.18.5

[tim-allen-ck]

We have come across this before, one approach is to create an allow list of packages on Nexus, so that only approved tools can be installed.

We will do some investigation to see if this can be blocked at the network level.

[TonyWildish-BH]

Blocking packages at Nexus won't solve this problem, it's easy enough to get the necessary client set up to allow DNS tunneling by other means.

Cutting off network access completely is one option, but that will make the TRE too hard to use for most users.

We need a solution that blocks this at the network level. I thought Defender for DNS would do this, but it seems not to exist anymore.

[tim-allen-ck]

This might be an option but it needs some time to look into. https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/what-s-new-introducing-microsoft-sentinel-dns-essentials/ba-p/3803389

[jonnyry]

We tried enabling Defender for DNS, which it turns out doesn’t exist anymore, so I enabled Microsoft Defender for Servers Plan 2 instead, which we thought might give us the same protection. It doesn’t, he was still able to exfiltrate data.

Was it perhaps sending alerts, rather than straight out blocking the connection?

There's some documentation here for it although its rather basic: https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-dns-introduction

[TonyWildish-BH]

@jonnyry , Defender for DNS appears to be deprecated, but the documentation is stale, yes. With Defender for Servers Plan 2 we do get alerts, so we can react, but not prevent the exfiltration from happening in the first place.

@tim-allen-ck , thanks for the link, but that appears to be monitoring, not prevention?

[marrobi]

The only way I am aware of to prevent DNS tunnelling, is to have a DNS server with an allow list of domains.

There is no Azure service that I am aware of that does this is out of the box.

One possible solution would be to use an Azure DNS Private Resolver, with a wildcard to route unknown domain names to a bad IP. Not sure if this would work, but worth looking at.

Another option would be a solution like Adguard Home (not sure if there are other corporate solutions) that have a list of rules to allow/deny that support regex or similar.

The workspace template VNet would need to be amended to use the custom DNS server.

Ultimately there is a risk/usability discussion which comes down to the organisation stance on this. If think about "Safe People", you could also say the Airlock is an exfiltration risk as the Airlock Manager could approve the export of PII (intentionally or accidentally).

I do think having an allow list on Nexus would be a better default for the OSS project, but it has been debated in the past and we have left it to implementers of the solution to decide.

[SvenAelterman]

Indeed, in this case, both Defender for Servers and Sentinel would detect, but not prevent. They're still critically important aspects of a comprehensive risk reduction strategy. Additionally, the response to those alerts can be automated. For example, the offending IP can be completely blocked by the firewall.

It should be noted (and perhaps that's what Tony hinted at here) that even allow listing packages isn't going to cure all. I suspect you'll need to allow Python on the systems so that data analysis can be performed. I am not a Python developer, but I suspect a malicious insider could write Python code to perform DNS tunneling.

Configuring your EDR solution on the endpoints might be the next best step. The EDR might be able to prevent tools or maliciously code like that from executing.

Solutions could also include placing a third-party firewall with advanced DNS security upstream of the TRE firewall.

Marcus has provided several other feasible options.

What's the recommendation of your pen tester for blocking DNS tunneling?

[jemrobinson]

Shameless plug for a project I work on: The Alan Turing Institute Data Safe Haven which is also an Azure-based open-source TRE. We solve this problem by using Adguard Home to provide a DNS server inside the TRE which only allows connections to a pre-approved allowlist of domains.

[TonyWildish-BH]

thanks for the replies. I'm aware that there's a tradeoff, we need to decide our appetite for risk, that's always going to be the case, but that's not the only consideration.

We're deploying the TRE for use with very sensitive data, and we have a responsibility to show that we're taking reasonable steps to protect it. I'm not overly surprised that such a problem exists in the TRE, but I'm disappointed that such known limitations aren't documented anywhere, I would have expected to find this mentioned in the docs. That would have allowed us to look into a resolution earlier, or possibly into other solutions.

Are there any other major security holes that you know of that aren't documented?

@SvenAelterman, we don't have the full report from our pen tester yet, but it's clear we can't go live with this problem.

@jemrobinson, I'd love to have a chat about how you implemented your solution. I've sent you a contact request via LinkedIn, can we pick up the conversation from there.

[marrobi]

One solution could be to create an adguard home shared service.

Configure Vnets to use Azure DNS private resolver, which forwards to Azure Firewall as the DNS proxy, that proxies to ad guard.

Then if we find an alternative solution, or customers have a comercial offering, it's a case of just amending the Firewall proxy.

@SvenAelterman thoughts?

[marrobi]

Ok, think I've got an easier fix, maybe not as as pretty as adguard home, but if create a private dns resolver and ruleset, link it to the workspace VNet you can restrict what external domains can be resolved. It still resolves privatelink addresses before going to the list of rules - this is why need to use a private resolver.

So suggest create a resolver in core, needs a dedicated subnet for outbound traffic, and link to the workspace VNets.

[SvenAelterman]

@marrobi I agree, definitely a much more supported solution because you can send the diagnostic logs to the Log Analytics Workspace, it's Azure native, etc.

You probably still should have the spokes send their DNS queries via the Firewall proxy and the Firewall should use the DNS private resolver.

[marrobi]

Will the private endpoints attacked to the workspace VNet work that way?

[TonyWildish-BH]

@marrobi, thanks for the pointers. I'm doing a PoC for this, one thing I haven't figured out is how to configure the rules to use the Azure internal DNS servers. Any suggestions?

It all works fine pointing at 8.8.8.8, but I'd rather not use Google's DNS servers if I can use the internal Azure DNS servers instead.

[marrobi]

Absolutely - https://learn.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16

My open questions are:

1. Can we do this with a single Private DNS resolver in core, or does that break private DNs zones only linked to the workspace VNet, such as azure monitor.

2. How do we maintain list of allowed FQDNs, for example we use service tags for many of the workspace services. We likely need a set of top level domains but also the ability to add new ones as required. This could be similar to how firewall rules are added by workspace services.

[TonyWildish-BH]

thanks for the quick reply, @marrobi. I've seen that page, but I don't think it helps. I tried putting 168.63.129.16 instead of 8.8.8.8 in my rules, and it's forbidden, the portal won't let me do it. Have I missed something?

[marrobi]

Hmm. Maybe the resolver doesn't have access to the vnet. I can give it a go tomorrow. So you have a corporate DNS server? Maybe makes sense to use that. We could make it an option in the config.

@SvenAelterman might have some knowledge that helps.

[marrobi]

Might be need an outbound endpoint into a vnet. I'll try figure it out tomorrow.

[marrobi]

Been thinking about this more, if we have a private resolver in each workspace network, using the Firewall as a DNS proxy (requires standard SKU), does the firewall automatically block all other DNS traffic unless add a network rule for the FQDN? This would be easier to maintain as FQDNs already exist on the firewall.

https://learn.microsoft.com/en-us/azure/firewall/fqdn-filtering-network-rules

We need to look into the two options.

[marrobi]

168.63.129.16

So seems like this isn't allowed. If you enable the DNS proxy on the firewall, then point the DNS resolver rule to the Firewall IP, this works. As Sven says you can also enable DNS logging on the firewall to monitor the DNS requests.

Still looking to see if we can do this on the firewall itself so we get Service Tag support, rather than on the DNS resolver.

[marrobi]

Ok, have a configuration that works.

• Firewall deployed with standard sku
• Enable DNS proxy on the firewall
• Deploy DNS resolver with outbound endpoint in the core VNet (created a new subnet)
• Create new ruleset with allowed DNS suffix going to firewall IP and a all for . going to 0.0.0.0 (bit of a bodge, but works)
• Link ruleset to workspace VNet

This way can have a centrally managed list, suggest deploy as part of the firewall shared service as can be updated in a similar way with FQDNs required by services.

[TonyWildish-BH]

thanks Markus, I'll give that a go.

[marrobi]

Hi @TonyWildish-BH, did you successfully make the changes? Any plans to provide a PR back to the project? Thanks.

[TonyWildish-BH]

Hi @marrobi. Almost completed, now in final testing, I'll provide a PR when I'm sure it's working properly.

One thing I realised is that the whitelist can (probably) be completely empty, i.e. we can sinkhole everything. Then the VM can only communicate with the Azure services and other services in the TRE, such as Nexus etc. That seems to work in the limited testing I've done so far.

The only thing that would make that not true is if certain applications rely on being able to at least look up the name of a server. They may be able to survive not being able to connect to it, but may spit the dummy if they can't even resolve the name. I don't know of any that do that, but it's possible.

The nice thing about sinkholing everything is that I don't have to rely on the Google DNS servers for the whitelist, I haven't found a way to use MS DNS servers, as noted above.

[marrobi]

You can use the Firewall IP as the DNS server as DNS proxy is enabled, and that will in turn use Azure DNS.

Some DNS names will need to be able to resolve - for example to access Azure storage it will need to access public DNS which then returns a privately resolvable DNS name.

Maybe you can share a draft PR and we can test it out and offer feedback?