When recently working on an issue it was discovered that some bad HAproxy rules were all but one node to be forbidden from communicating with stanchion. There were no errors in the logs that could not communicate and bucket/user requests to nodes which could not communicate with stanchion simply returned a generic 503 error.
There should be further logging when a request is unsuccessful because of stanchion being unavailable or blocked for some reason. Additionally a CLI tool to check stanchion availability such as riak-cs-admin stanchion which returned true or false would be very helpful.
When recently working on an issue it was discovered that some bad HAproxy rules were all but one node to be forbidden from communicating with stanchion. There were no errors in the logs that could not communicate and bucket/user requests to nodes which could not communicate with stanchion simply returned a generic 503 error.
There should be further logging when a request is unsuccessful because of stanchion being unavailable or blocked for some reason. Additionally a CLI tool to check stanchion availability such as
riak-cs-admin stanchion
which returned true or false would be very helpful.