[Swarm] Research using "source-ip-affinity-mode" from Azure

[christopherdro]

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-distribution-mode#source-ip-affinity-mode

[christopherdro]

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-sticky-sessions.html

[cjcaj]

We can configure our service to use session affinity based on ClientIP but this might break with how we're proxying our ssl connections. https://kubernetes.io/docs/concepts/services-networking/service/

I'll need some time to test the new setup.

[cjcaj]

I've got session affinity working with cookies. I just have to clean up the cors settings a bit then we should be good to go.

[christopherdro]

Nice! Do we know how this will work with requests that aren't coming from a browser?

[cjcaj]

We'll have to base it on ClientIP instead. Since we can't use the ELB to terminate the connection to keep the source IPs intact, the ELB will proxy the connections as is and our pods will have to handle authentication themselves.

[christopherdro]

Getting consistent 404's now. Even on old pastes that were created last week. Any ideas what's up?

[cjcaj]

My pastes are taking 30 seconds to load. Have you tried waiting for the request to test.network.bzz to finish? I did notice that the first request hits cloudfront with the hash and gets a 404 and cloudfront is probably serving the index.html. We could probably fix that by putting the hash as a query param.

https://blockpaste.com/7aeadb46e5d2e4c44ee43aa5e1b2e23481e9200dac7b110fc54baff1d6abec67#U19HEcHAekexiYwrsvGm13Ov0nkUMAr9hMYRV23WsO0=

https://blockpaste.com/dc601b69c723fb620a849286e6e18b98f1f96f485fddec5bc719c137edd3adc9#OdJdfs6P57J11xZbDPS8RGTIKwIoptORjPXUZJvV4Tc=

I'll look into why it's taking so long. Even new pastes are taking 30s to fetch.

[christopherdro]

Are old pastes working for you?

[cjcaj]

Yes, the second link is 4 days old. Can you link your old pastes here?

[cjcaj]

I had forgotten that I had the geth pod in a pending state to test out AlertManager. Now it's back up and requests are back to normal ~300ms. The swarm pods must have been trying to reach geth for ens and it timed out in 30 seconds.

[christopherdro]

Nice. Outside of 404 I'm getting a cors issue from one of the pods.

Failed to load https://test.bzz.network/bzzr:/cedf89a4dc45930b651f74847c20a8b068b2fc80358b8929659575eb32c6a979: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://blockpaste.com' is therefore not allowed access. The response had HTTP status code 404.

Do we have it set to wildcard on both pods?

[cjcaj]

The CORS headers will be set by ingress and not as the swarm flag. It's not adding the headers since the response was a 404. We could change this so that the headers are set for any status code.

[cjcaj]

404s respond with CORS headers now.

[christopherdro]

Shouldn't we no longer be getting 404's now that we have session affinity working with cookies?

[cjcaj]

I used a bad hash for testing. It also looks like the link you posted earlier is returning 200 now.

[christopherdro]

The main issue still persist of getting a 404 right after creating a new paste.

[cjcaj]

Here's a new paste that returns 200

POST response:

First GET response:

Second GET response (refresh the page):

You can see that the SetCookie route for the responses are: POST: 00de5d13203bcd0d08144469997c9105d38fdd93 GET 1: 7c23b0257a11de8d629f0208b910e6a291841326 GET 2: 00de5d13203bcd0d08144469997c9105d38fdd93

So the sticky sessions didn't work because we never merged the PR https://github.com/async-la/blockpaste/pull/32. But the good news is that the pods have set each other as peers and it doesn't matter who you make the request to! 🥂

[cjcaj]

But if you're still getting a 404, could you post the response headers like above with the POST, GET 1, and GET 2 (refresh) requests

[christopherdro]

💯 💯 💯

Working!