wss in not being upgraded on AVI SSL-reencryption.

[mjudeikis]

We get ws 403 errors on console with user token

Version

Server https://console.int:443 openshift v3.3.0.35 kubernetes v1.3.0+52492b4

Steps To Reproduce

1. We run AVI load balancer as our LB for Openshift.
2. Create VIP with static DNS record and pool members are set Masters
3. Session affinity is set to client IP

p.s Token is valid.

Current Result

Expected Result

Session is kept alive.

I dont expect solution, but maybe you seen this somewhere else or know how to debug it...

[jwforres]

Is it possible something in your load balancer is stripping query params off the request?

@liggitt any thoughts on this?

[jwforres]

or @deads2k since @liggitt is OOO

[mjudeikis]

We do SSL termination on AVI VIP. So we still suspect it may be issue on How AVI Load Balancer terminates SSL... Still bumpt

[deads2k]

Is it possible something in your load balancer is stripping query params off the request?

LoadBalancer is the first place I would look. I'm not a wss expert, but you could try mocking up a call to oapi/v1/users/~ through the websocket to see which user the openshift api server thinks you are.

[mjudeikis]

I checked Token ownership and token is same one as in the UI session.

[deads2k]

Calling the user/~ endpoint would let you know if the server is getting your token and you simply don't have rights. That would be unusual, but possible if something is missing in your roles.

[jwforres]

From the network trace in the initial screenshot I can see the token was valid, the regular https requests were succeeding, only the wss requests were failing.

On Tue, Dec 13, 2016 at 9:30 AM, David Eads notifications@github.com wrote:

Calling the user/~ endpoint would let you know if the server is getting your token and you simply don't have rights. That would be unusual, but possible if something is missing in your roles.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/openshift/origin/issues/12222#issuecomment-266752514, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZk7eVIEmm59UZpjAOO-rPyLQpHZn24ks5rHqvrgaJpZM4LKl50 .

[mjudeikis]

Yes. The difference is that for HTTP you use in the Header:

Authorization:Bearer GalvMr6mknly-SOslIj0FjXg_6nqZLSf0ZcNF4kyofQ

and for WSS:

watch:true
access_token:GalvMr6mknly-SOslIj0FjXg_6nqZLSf0ZcNF4kyofQ

My JS skills are rusty but im trying to recreate this with simple:

....
var wsUri = "wss://console.int.xxx/oapi/v1/projects?watch=true";  
websocket = new WebSocket(wsUri,  ["access_token", "GalvMr6mknly-SOslIj0FjXg_6nqZLSf0ZcNF4kyofQ"]);
...

And getting undefined. will try dig into LB more and if found smth will drop here :/

[mjudeikis]

Got involved with AVI. So apparently Openshift never responds with "upgraded" WS connection and instead gives 403.

Tried checking Logs on api by increasing log level to 4 (/etc/sysconfig/atomic-openshift-master-api) but cant see any request being logged.

Where does Logs for oapi/api on backed being logged?

[deads2k]

Where does Logs for oapi/api on backed being logged?

Take a look here: https://docs.openshift.org/latest/install_config/master_node_configuration.html#master-node-config-audit-config . Enabling the audit logging will show every connection.

[mjudeikis]

failing setup: AVI Loadbalancer is presenting certificates and doing SSL-ReEncryption. During SSL reencryption backend does not recognize token/cookie as valid and gives 403. This is why session is never upgraded to WS.

Fix: We removed certificates from AVI and placed them on Openshift/Kubernetes itself and reconfigured AVI tp be L4/Passthrouth loadbalancer. We tried same thing with HAproxy re-encryption, and everything worked out of the box.

Question: Do you do any session pinning based on SSL certificate? or anything under these lines?

Closing this one.

[mjudeikis]

Fixed in AVI 16.3.4.