search

nucypher / nucypher

Threshold Access Control (TACo) runtimes for the Threshold Network
GNU Affero General Public License v3.0
691 stars 271 forks source link

Running TACo as HA reverse proxy backend #3470

Open sammy007 opened 5 months ago

sammy007 commented 5 months ago

Please clarify is it normal for network if TACo node will be working under a reverse proxy which works in its own SSL mode so peers will get 9151 endpoint using a snake oil certificate which is different than TACo's snake oil certificate.

sammy007 commented 5 months ago

And a follow up question, is it fine if TACo will not see real IP addresses of the clients making rest requests?

sammy007 commented 5 months ago

Please teach Ursula to respect X-REAL-IP. It's just impossible to put it behind a balancer in one way or another. There is literally no reason to not try to extract it if it needed so much.

derekpierre commented 5 months ago

Because nodes participate in rituals and need to store cryptographic material, load-balanced nodes would need to share disk state. Also, there is the issue of self-signed certificates generated by the node, and a new one is generated on each restart. You'd need some customization to accommodate that.

I think (?) some people have put their nodes behind a proxy, so with sufficient configuration on your side, you can perhaps make it work.

Load-balancing capabilities is not currently high on the priority order.

derekpierre commented 5 months ago

Please teach Ursula to respect X-REAL-IP. It's just impossible to put it behind a balancer in one way or another. There is literally no reason to not try to extract it if it needed so much.

This was already handled in v7.1.0 - see https://github.com/nucypher/nucypher/pull/3398

sammy007 commented 5 months ago

This was already handled in v7.1.0

Doh, got it.

and a new one is generated on each restart. You'd need some customization to accommodate that.

It would be nice if Ursula will stop regenerating SSL certificate on each restart, does not make sense, but what I know. I just can't find a specific reason for Ursula to not generate it once based on the seed from init phase and let it survive restarts. This way if somebody wanna put Ursula behind a reverse proxy that one can take that certificate and terminate with it + add XFF header. Otherwise it's impossible to balance with Layer7, only plain TCP passthrough which removes the ability to pass XFF header.

derekpierre commented 4 months ago

It would be nice if Ursula will stop regenerating SSL certificate on each restart, does not make sense, but what I know. I just can't find a specific reason for Ursula to not generate it once based on the seed from init phase and let it survive restarts. This way if somebody wanna put Ursula behind a reverse proxy that one can take that certificate and terminate with it + add XFF header. Otherwise it's impossible to balance with Layer7, only plain TCP passthrough which removes the ability to pass XFF header.

The reason for regenerating the cert on restart is to simplify the need for users to need to check whether the cert has expired or not, or to forcibly change the cert if the IP address changes. It was simpler than finding ways for users to proactively monitor and update the cert. See https://github.com/nucypher/nucypher/pull/2701, https://github.com/nucypher/nucypher/issues/2674.

There are users who have run Ursula behind a proxy - https://github.com/nucypher/nucypher/issues/3378.

sammy007 commented 4 months ago

Load balancing is a wide subject and does not always mean there are N active instances at a time.

The deployment scenario could be as follows: N nodes with N block level replicated disks. One active node at a time, N disk copies always. When node X fails you have the same state at the time node X went down and Ursula on node Y starts.

As to IP address, does it really matter that same disk state (including ursula key) but with different IP address requires different SSL certificate? I don't think so. It's basically the copy of process just serving from different IP address. Other options is to put ALL these active and stand by nodes behind highly available load balancer which requires TLS/SSL termination (Layer7 because it will be impossible to set X-Forwarded-For without unwrapping request on HTTP level). Ursula TLS certificate could be cached in the same directory where config file persists (not on init phase, just on start phase if it does not exist or does not match seed) so one can take this certificate and configure HA proxy with it so the requests will be valid when fail over happens and (or not) the IP address is the same. Also, the X-Forwarded-For will be specified so Ursula will have real IP of the client as that HA proxy is a trusted instance.

As to X-Forwarded-For I put arbitrary IP address 1.1.1.1 to X-Forwarded-For and requested /ping without configuring anything on Ursula side and got a reply with 1.1.1.1 which leads to understanding that Ursula in fact accept this header without really being configured to do that (without being behind trusted proxy) and ANY client can forge IP address. I have no idea if that is big issue, I just think it makes sense to highlight it.

sammy007 commented 4 months ago

Well, wanna just leave it here for future when there will be more time available, it seems there is out of the box wrapper in twisted for the issues I tried to highlight, https://docs.twistedmatrix.com/en/twisted-21.7.0/api/twisted.protocols.haproxy._wrapper.HAProxyProtocolWrapper.html

Also for reference: proxy protocols

KPrasch commented 4 months ago

Well, wanna just leave it here for future when there will be more time available, it seems there is out of the box wrapper in twisted for the issues I tried to highlight, https://docs.twistedmatrix.com/en/twisted-21.7.0/api/twisted.protocols.haproxy._wrapper.HAProxyProtocolWrapper.html

Also for reference: proxy protocols

Good find - Twisted never ceases to surprise me with it's comprehensive network tools. I'll have a closer look when we can carve out the time 👍🏻

sammy007 commented 4 months ago

Good find - Twisted never ceases to surprise me with it's comprehensive network tools. I'll have a closer look when we can carve out the time 👍🏻

Yeah very neat.

The proxy protocol support can remove the requirement of unwrapping TLS on HA frontend and providing that certificate to a DC admins in order to set up balancing because simple layer4 could be requested with proxy protocol enabled. As to TLS thing, I still believe there is no reason for a volatile TLS certificate in ursula, it could perfectly be cached in a directory along with a config to survive restarts and recreated if cached certificate does not exist (not on init phase, on just normal restart) as it depends on a seed from init phase.

Thanks.