Backend caching returns 500 when apisonator is down

[pehala]

When Backend caching is enabled and apisonator is down, uncached credentials return always 500 regardless of BACKEND_CACHE_POLICY_FAIL_CLOSED. Cached credentials work as expected.

[unleashed]

Thoughts from a preliminary inspection:

• The code applies a policy and just fills in the response to send back to the client with the actual reason the last connection to 3scale failed (ie. 503 service unavailable) rather than a 403/Permission Denied. This happens however policy is configured, but with the open policy it will try to connect again (and unless something changed, it will get back a 50X status), but it will still be the "last connection status code". So if 3scale is always unreachable, setting this policy to true/false will appear to make no difference.
• Even though WithUnavailable (protobuff rpc mapping for 503 Service Unavailable) is actually present in the Istio Mixer's status package in the vendored dependency tree, it seems not to be available in the latest releases I can see in the upstream repos. This is also the case with a few other cases. I don't know why this changed happened, and my speculation is since it has been removed from there, you can't possibly send such a status code to the latest Mixer with the latest protobuff definition. Instead, and given how many such funcs end up in 500 Internal Server Error, my thinking is Mixer will just interpret it as a deprecated message and use instead a 500 status code.
• I can't even find an updated rpc package definition (404 Not Found) as imported in either the latest mixer sources nor the vendored dependency tree.
• If no matching With* status function for the returned HTTP status code from 3scale is available, WithStatusUnknown is used instead, which ends up becoming a 500 Internal Server Error.

So from a quick check I think this might need to change in the fail_closed case to return a 403 instead of relaying back the code that 3scale last failed with. The question about why this 503 -> 500 mapping is happening looks less important, and probably not worth getting into.