invalid routes gloo config not honored

[mbarlow]

Describe the bug

Following the instructions here: https://docs.solo.io/gloo/latest/guides/traffic_management/configuration_validation/invalid_route_replacement/ (modify for rio-system namespace) does not result in invalid route handling by gloo. Confirmed settings for "replaceInvalidRoutes" is set to true by doing:

kubectl get settings -n gloo-system default -oyaml

To Reproduce

1. install rio w/ clusterdomiain & deploy a workload. Verify accessing your workload.

2. Patch gloo settings. kubectl patch settings -n rio-system default --patch '{"spec": {"gloo": {"invalidConfigPolicy"{"replaceInvalidRoutes": true}}}}' --type=merge

3. Modify workload URL and specify a path that does not exist on workload, or just hit a fictional sub/subdomain on your domain.

4. See that response body is not the value from gloo settings. If changing response code, see that it is not honoring value. 404 returns. Response body is empty

Expected behavior Response body for any bad routes will be replaced with settings value in gloo.

Kubernetes version & type (GKE, on-prem): kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2+k3s1", GitCommit:"cdab19b09a84389ffbf57bebd33871c60b1d6b28", GitTreeState:"clean", BuildDate:"2020-01-27T18:09:26Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Type: Rio version: rio info

 v0.8.0-rc1 

Additional context rio system logs output:

[StrongMonkey]

@mbarlow Can you try with the latest rc2? In this release I have updated gloo to the latest release. Looks like a bug on their side.

[mbarlow]

@StrongMonkey upgraded to rc2. Retried same steps. I'm still not getting the body response from settings on 404