Replace SRE Azure Firewall with cheaper alternative

[jemrobinson]

:white_check_mark: Checklist

• [x] I have searched open and closed issues for duplicates.
• [x] This is a request for a new feature in the Data Safe Haven or an upgrade to an existing feature.
• [x] The feature is still missing in the latest version.
• [x] I have read through the documentation.
• [x] This isn't an open-ended question (open a discussion if it is).

:strawberry: Suggested change

The Azure Firewall is moderately expensive (~£200/month) and we're only using basic filtering capabilities. We should look at alternatives.

:steam_locomotive: How could this be done?

We should look into replacing this with a Squid Proxy - @manics might be able to give some advice on how TREEHOOSE do this.

[manics]

Our Ansible roles for configuring our EC2 proxy VMs are open-source: https://github.com/hic-infra/shared-services-ansible-roles/

We're in the process of open-sourcing the Terraform for deploying the EC2 instances and everything else for our shared services, though at some point we may switch to containerising everything.

[jemrobinson]

Config examples here:

https://blog.thinkbox.dev/posts/0009-domain-filter-with-squid/

https://wiki.squid-cache.org/SquidFaq/SquidAcl

https://xebia.com/blog/how-to-configure-squid-as-an-egress-gateway/

https://jasonpangazure.medium.com/how-to-use-azure-firewall-and-squid-as-virtual-appliance-in-azure-route-table-to-overwrite-debc98b8f0b8

[JimMadge]

That looks very nice,

• Would save costs if we can remove Azure firewall.
• Plain text configuration is nice (potentially there is a Pulumi provider or Ansible module for squid, if not, writing our own class probably isn't too bad).
• Would need to make sure if if Squid goes down all outbound/inbound connection is lost.

[manics]

If you're writing your own Ansible role you might be able to copy/fork https://github.com/hic-infra/shared-services-ansible-roles/tree/main/squid

[jemrobinson]

It looks like getting Squid to work with HTTPS is complicated (see e.g. https://dev.to/suntong/squid-proxy-and-ssl-interception-1oa4) and is likely to involve installing a self-signed certificate on all resources that need to make HTTPS connections. I don't think this will work with calls from Dockerised services.

[JimMadge]

HTTPS in general is tricky. Can't use the simple cert verification challenges if you have no internet access. Can't reach cert authorities to check certs with no internet access.

HTTPS inside would be nice to have. I don't have much experience with self-signed certs. I think a lot of programs reject or raise warnings about them. It might be difficult to add and trust the certs everywhere they are needed.

[jemrobinson]

This is less about accessing e.g. https://gitea.<my sre>.com inside the environment but more about accessing https://login.microsoftonline.com for user authentication.

[JimMadge]

In both cases do you get the problem of, the squid server doesn't have a valid cert for the domain you are requesting. It would look like a MITM attack.

[jemrobinson]

I mean, it is a MITM attack. The proxy is essentially unwrapping an HTTPS request to find its destination, deciding whether or not to forward it on, making a new request, getting the result of that request and sending it back to the original client.

[JimMadge]

But the way we propose it is a friendly attack :smile:.

HSTS might also make it difficult to do in a browser. The browser will know certain sites should always be served over HTTPS.

[jemrobinson]

NB. Azure Firewall does this by resolving FQDNs to a list of IP addresses every 15 seconds (https://learn.microsoft.com/en-us/azure/firewall/fqdn-filtering-network-rules#how-it-works). Could be a way forward if we're happy to write some code to do that?