401 errors when getting MITM'ed

Okay, so a bit of a weird one that I had to poke my toe into the rabbit hole to find.

Bit of background first - my current company intercepts some traffic on a corporate device while it is not connected to a VPN - probably for some deep packet inspection of some variation.

When trying to create a new cluster within rancher, while not on a VPN (which bypasses this), we get a 401 Authentication error from rancher.

With an basic tf config

rancher2_project.test: Creating...
2020/12/23 23:32:25 [DEBUG] EvalApply: ProviderMeta config value set
2020/12/23 23:32:25 [DEBUG] rancher2_project.test: applying the planned Create change
2020/12/23 23:32:25 [TRACE] GRPCProvider: ApplyResourceChange
2020-12-23T23:32:25.583Z [DEBUG] plugin.terraform-provider-rancher2_v1.10.6: 2020/12/23 23:32:25 [DEBUG] setting computed for "annotations" from ComputedKeys
2020-12-23T23:32:25.583Z [DEBUG] plugin.terraform-provider-rancher2_v1.10.6: 2020/12/23 23:32:25 [DEBUG] setting computed for "labels" from ComputedKeys
2020-12-23T23:32:25.583Z [DEBUG] plugin.terraform-provider-rancher2_v1.10.6: 2020/12/23 23:32:25 Getting from  https://rancher-domain.com/ping
2020-12-23T23:32:26.184Z [DEBUG] plugin.terraform-provider-rancher2_v1.10.6: 2020/12/23 23:32:26 Time to get req:  601  ms
2020/12/23 23:32:26 [DEBUG] rancher2_project.test: apply errored, but we're indicating that via the Error pointer rather than returning it: Bad response statusCode [401]. Status [401 Unauthorized]. Body: [message=Unauthorized 401: must authenticate] from [https://rancher-domain.com/v3]
2020/12/23 23:32:26 [TRACE] EvalMaybeTainted: rancher2_project.test encountered an error during creation, so it is now marked as tainted

However, this does work while not getting MITM'ed.

Things of note:

The MITM uses a different TLS cert than what rancher is configured with
I'm currently using github auth on the frontend, however tf is configured to use a bearer token - I don't think this makes any difference
Activation of the MITM is quite random - however it seems like attempting to directly to initiate a connection via IP without first querying DNS can cause issues? DNS TTL's are set to 5 secs,

I couldn't see anything that stood out in the rancher logs. Weirdly, I could see the ping request in the audit log, but there wasn't any record of attempts to v3 when attempts were successful?

{"auditID":"d5dcf952-d4a9-420d-9e0c-4aa6d1d20c83","requestURI":"/ping","user":{"name":"system:cattle:error","group":["system:unauthenticated","system:cattle:error"]},"method":"GET","remoteAddr":"192.168.0.1:61629","requestTimestamp":"2020-12-24T00:05:39Z","responseTimestamp":"2020-12-24T00:05:39Z","responseCode":200,"requestHeader":{"Accept-Encoding":["gzip"],"Referer":["https://rancher-domain.com.x.12345678900af710397ce1f0b767.1230fc45.id.opendns.com/s/rancher-domain.com/ping?X-OpenDNS-Session=_1234567890960af710397ce1f0b7679270fc45_cGLTfZMd_"],"User-Agent":["Go-http-client/1.1"],"X-Forwarded-For":["192.168.0.1"],"X-Forwarded-Host":["rancher-domain.com"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Proxyuser-Ip":["123.123.123.123"],"X-Real-Ip":["192.168.0.1"],"X-Request-Id":["12345678907fda6afb24f9ab5ed63"],"X-Scheme":["https"]},"responseHeader":{"X-Content-Type-Options":["nosniff"]}}
{"auditID":"0374bb00-9217-42e2-a189-ad8ea48247ac","requestURI":"/v3","user":{"name":"system:cattle:error","group":["system:unauthenticated","system:cattle:error"]},"method":"GET","remoteAddr":"192.168.0.1:61629","requestTimestamp":"2020-12-24T00:05:40Z","responseTimestamp":"2020-12-24T00:05:40Z","responseCode":401,"requestHeader":{"Accept-Encoding":["gzip"],"Referer":["https://rancher-domain.com.x.12345678908087060cd7b55f77c9c.1234fc46.id.opendns.com/s/rancher-domain.com/v3?X-OpenDNS-Session=_12345678907b55f77c9c9270fc46_35MyMNy7_"],"User-Agent":["Go-http-client/1.1"],"X-Forwarded-For":["192.168.0.1"],"X-Forwarded-Host":["rancher-domain.com"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Proxyuser-Ip":["123.123.123.123"],"X-Real-Ip":["192.168.0.1"],"X-Request-Id":["1234567890f6a3add18d18cb1982ef4"],"X-Scheme":["https"]},"responseHeader":{"Content-Type":["application/json"],"X-Content-Type-Options":["nosniff"]},"responseBody":{"type":"error","status":"401","message":"Unauthorized 401: must authenticate"}}

I guess this is fairly out of scope, and isn't really a bug with the provider. I guess it would be cool to hear some thoughts on what could cause this, and if it's solvable

terraform 0.14.3 rancher2 provider 1.10.6 rancher 2.5.3

rancher / terraform-provider-rancher2

401 errors when getting MITM'ed #553