search

hyperledger / besu

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client https://wiki.hyperledger.org/display/besu
https://www.hyperledger.org/projects/besu
Apache License 2.0
1.52k stars 847 forks source link

Discovery using Kubernetes #7013

Open daanporon opened 7 months ago

daanporon commented 7 months ago

Description

I'm trying to get Discovery to work on a Kubernetes environment using a LoadBalancer and the Kubernetes NAT manager. And i'm experiencing multiple issues with this. So now i'm wondering if i'm doing something wrong or that it wasn't the intention to use Discovery in this way. I know the documentation says that there are limitations, but i was trying to see if i can work around those and maybe propose fixes for them.

I'm setting up my nodes using Pulumi scripts. I also have dns enabled and my nodes are accessible via a dns name.

What i do to test this.

How i set up a node:

Things i already figured out:

Issues i'm now still experiencing:

I have been testing this on both Azure and on AWS, because GKE doesn't support these kind of mixed-loadbalancers right now.

Possible fixes:

daanporon commented 6 months ago

Keep sending the From data in the PacketData so that it never falls-back to the source of the packet.

I don't think we need to do this for every type of packet, but only the ones where you are the initiator. So i think PING, ENR_REQUEST and FIND_NEIGHBOURS?

daanporon commented 6 months ago

Would be good to get some feedback, so maybe i can try to implement those changes ... but i'm not familiar enough with all of the code to make sure it are acceptable solutions.

daanporon commented 6 months ago

Maybe in stead of xdns-domain-name we should allow p2p-host to be a domain-name if xdns-enabled=true?

joshuafernandes commented 6 months ago

Hi @daanporon I think this is a really good idea. For AWS though, you don't specifically need a loadbalancer and can skip that part out. I've got a PR https://github.com/hyperledger/besu-docs/pull/1597 here you can use which makes use of ec2 instances directly to establish connectivity. Haven't found an equivalent for Azure yet so this would be a good solution. @matkt is the best person to ask about the NAT manager

daanporon commented 5 months ago

We now also did something similar, using NodePorts services and using the ip address of the nodes where are containers are hosted on. Tested this in AWS and GKE and seemed to work find. We are using kubectl get nodes to fetch the external ip of the node, which can be used generically across cloud providers.

joshuafernandes commented 5 months ago

Good to hear @daanporon ! I''m working on some charts for besu and teku that can be used with the above implementation that I'll make available soon. nodeport is fine too for one/few nodes, but if you have many you can't reuse the same service across host nodes as there is port contention. I've used the clusterip to overcome that and kept the RBAC of the pods to absolute least privileges. Either way though am happy you have a working solution :)

The cloud providers also use a metadata service to return the IP so that is another option (well I know AWS and Azure do, and I think GKE do the same as well)

daanporon commented 1 month ago

@joshuafernandes where you able to get it working without the PR i did here? https://github.com/hyperledger/besu/pull/6578

Because i can only get it to work if i patch my Besu network i'm connecting to. Because if i don't do this it will try to send the PONG data to the wrong port and possible IP, which wasn't opened on my Kubernetes cluster.

joshuafernandes commented 1 month ago

Hi @daanporon I wound up just keeping it simple and using an external service to return the IP https://github.com/Consensys/ethereum-helm-charts/blob/main/charts/ethereum-helm-charts/charts/elc/templates/besu/statefulset.yml#L66 We bind directly to the host node's IP and steps to get things setup on the infra side are here https://besu.hyperledger.org/public-networks/tutorials/kubernetes

This still keeps the RBAC permissions intact and doesn't require admin rights which I prefer. Control goes back to the owner and they decide how/what is allowed

daanporon commented 1 month ago

I'm still not sure how this can work. Because if you see, this is the communication i get using NodePorts on the current stable besu release:

besu {"timestamp":"2024-10-04T09:07:29,904","container":"besu5npn1-01375-5bb9db8d9f-gnwnw","level":"INFO","thread":"main","class":"DefaultP2PNetwork","message":"Enode URL enode://ae75ff2eb7855dae17bf837c8537159495231c504942c0450924de56bd9643b16ff9caec0a10982057614f794e56d3c04ba5e879b9aca32f95712a5eb3e9ad09@3.66.85.127:32130","throwable":""}
besu {"timestamp":"2024-10-04T09:07:29,994","container":"besu5npn1-01375-5bb9db8d9f-gnwnw","level":"TRACE","thread":"vert.x-eventloop-thread-0","class":"DiscoveryProtocolLogger","message":"<<< Sending PING  packet to peer 0xab1636f5735f155c... (enode://ab1636f5735f155c0ea032a926bb2dd0030df383ebc545deeefc25782e4c69d6d4edb5684df16b377dcd50171028e0bb8be19fe4cea8392e44435f7ee9ba5fc8@besu5n1-36bb5p.eks-europe.settlemint.com:30303?discport=40404): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[3.66.85.127]', udpPort=32130, getTcpPort=32130}, to=Endpoint{host='Optional[3.121.248.224]', udpPort=40404, getTcpPort=30303}, expiration=1728032909, enrSeq=1}, hash=0x3a1f4c6c7abb1583d22b028d812b31f0d21cc8413850baf744f526470ac5a4c7, signature=Signature{r=79301454547643036649981252443536507601482132134448579096296194526647061090041, s=43816660833750126562122072462348771578582935481220187846693421630451543337435, recId=0}, publicKey=0xae75ff2eb7855dae17bf837c8537159495231c504942c0450924de56bd9643b16ff9caec0a10982057614f794e56d3c04ba5e879b9aca32f95712a5eb3e9ad09}","throwable":""}
besu {"timestamp":"2024-10-04T09:07:30,441","container":"besu5n1-36bb5-d58f6868-nthr9","level":"TRACE","thread":"vert.x-eventloop-thread-0","class":"DiscoveryProtocolLogger","message":">>> Received PING  packet from peer 0xae75ff2eb7855dae... (enode://ae75ff2eb7855dae17bf837c8537159495231c504942c0450924de56bd9643b16ff9caec0a10982057614f794e56d3c04ba5e879b9aca32f95712a5eb3e9ad09@3.66.85.127:32130?discport=14553): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[3.66.85.127]', udpPort=32130, getTcpPort=32130}, to=Endpoint{host='Optional[3.121.248.224]', udpPort=40404, getTcpPort=30303}, expiration=1728032909, enrSeq=1}, hash=0x3a1f4c6c7abb1583d22b028d812b31f0d21cc8413850baf744f526470ac5a4c7, signature=Signature{r=79301454547643036649981252443536507601482132134448579096296194526647061090041, s=43816660833750126562122072462348771578582935481220187846693421630451543337435, recId=0}, publicKey=0xae75ff2eb7855dae17bf837c8537159495231c504942c0450924de56bd9643b16ff9caec0a10982057614f794e56d3c04ba5e879b9aca32f95712a5eb3e9ad09}","throwable":""}
besu {"timestamp":"2024-10-04T09:07:30,471","container":"besu5n1-36bb5-d58f6868-nthr9","level":"TRACE","thread":"vert.x-eventloop-thread-0","class":"DiscoveryProtocolLogger","message":"<<< Sending PONG  packet to peer 0xae75ff2eb7855dae... (enode://ae75ff2eb7855dae17bf837c8537159495231c504942c0450924de56bd9643b16ff9caec0a10982057614f794e56d3c04ba5e879b9aca32f95712a5eb3e9ad09@3.66.85.127:32130?discport=14553): Packet{type=PONG, data=PongPacketData{to=Endpoint{host='Optional[3.66.85.127]', udpPort=14553, getTcpPort=32130}, pingHash=0x3a1f4c6c7abb1583d22b028d812b31f0d21cc8413850baf744f526470ac5a4c7, expiration=1728032910, enrSeq=3}, hash=0xc72099a541609ced5cc4ecd529a383c0ee9e43dc5ad0ffdb5a9438f6fdd0b09a, signature=Signature{r=11066360307279832640058640885693309553875085973167258113212992390556650576315, s=31389525697801267030296335889283349481952924453013161562625547003800422495047, recId=0}, publicKey=0xab1636f5735f155c0ea032a926bb2dd0030df383ebc545deeefc25782e4c69d6d4edb5684df16b377dcd50171028e0bb8be19fe4cea8392e44435f7ee9ba5fc8}","throwable":""}

As you can see it isn't using the advertised UDP Port here, it's using the one from the connection. But in my case i didn't open that port in the firewall, so it cannot connect to my node on my Kubernetes environment. This is what my PR is fixing, another example with my PR and the advertised UDP port opened on the firewall:

besu {"timestamp":"2024-10-03T14:08:37,244","container":"besu3npn1-5d8ba-84559f59bf-ndbph","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":"<<< Sending PING  packet to peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@besu3n1-3c094p.eks-europe.settlemint.com:30303?discport=40404): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, to=Endpoint{host='Optional[18.198.144.187]', udpPort=40404, getTcpPort=30303}, expiration=1727964577, enrSeq=3}, hash=0x8aafc14f2a94f5450437790117428d6e26cc4c60366bbd6b00d5a587e1627bf9, signature=Signature{r=21613984819245652446412272794611917349175691367434421316380350610881204600914, s=34346252345397575908378526518023857623531316063207757552808085299072876158512, recId=1}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:37,745","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":">>> Received PING  packet from peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:31666): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, to=Endpoint{host='Optional[18.198.144.187]', udpPort=40404, getTcpPort=30303}, expiration=1727964577, enrSeq=3}, hash=0x8aafc14f2a94f5450437790117428d6e26cc4c60366bbd6b00d5a587e1627bf9, signature=Signature{r=21613984819245652446412272794611917349175691367434421316380350610881204600914, s=34346252345397575908378526518023857623531316063207757552808085299072876158512, recId=1}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:37,747","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":"<<< Sending PONG  packet to peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:31666): Packet{type=PONG, data=PongPacketData{to=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, pingHash=0x8aafc14f2a94f5450437790117428d6e26cc4c60366bbd6b00d5a587e1627bf9, expiration=1727964577, enrSeq=3}, hash=0x05e89854f05fc450241bd5017a4e53569eec9d1c4d52c5df671fa04c7af87312, signature=Signature{r=54013045116450348155882834488072549433794517379861119449233919458872950577054, s=55784129346942186953562773115041573641854757700286365905433551089390144919820, recId=0}, publicKey=0x15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:42,269","container":"besu3npn1-5d8ba-84559f59bf-ndbph","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":">>> Received PONG  packet from peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@10.0.10.249:47715): Packet{type=PONG, data=PongPacketData{to=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, pingHash=0xd019f38d17920f19d9ecd60e2a936ab16cec99d044ba25ddb495757945955f46, expiration=1727964582, enrSeq=3}, hash=0x9c1591e4140ed60d4b8066e6827f30aa2fcb784e57d1e9ee77d343fa638f4b9b, signature=Signature{r=17785939030391684557475502496153583832484851596698797093383015501198293114849, s=25617559921057012432170844301872499027982248231665253708672723028901260650890, recId=1}, publicKey=0x15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:42,275","container":"besu3npn1-5d8ba-84559f59bf-ndbph","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":
"DiscoveryProtocolLogger","message":"<<< Sending ENRREQ packet to peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@besu3n1-3c094p.eks-europe.settlemint.com:30303?discport=40404): Packet{type=ENR_REQUEST, data=ENRRequestPacketData{expiration=1727964582}, hash=0xc5f283e094a08dd600b405fd2750aa4f82f1828f04ceb5410077a477daf91c16, signature=Signature{r=64495488120197147305131926293280176153810319318936811393307381748691035240952, s=3180036167744700803863625742721598557275798091649992309214594256056114751346, recId=0}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:42,279","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":">>> Received ENRREQ packet from peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:33513): Packet{type=ENR_REQUEST, data=ENRRequestPacketData{expiration=1727964582}, hash=0xc5f283e094a08dd600b405fd2750aa4f82f1828f04ceb5410077a477daf91c16, signature=Signature{r=64495488120197147305131926293280176153810319318936811393307381748691035240952, s=3180036167744700803863625742721598557275798091649992309214594256056114751346, recId=0}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:42,747","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":"<<< Sending PING  packet to peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:31666): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[18.198.144.187]', udpPort=40404, getTcpPort=30303}, to=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, expiration=1727964582, enrSeq=3}, hash=0xb5973fb98e7366954dd0b16c10b1b96d53ea383e5ae3dd80a7e81a847a69e997, signature=Signature{r=15969048044140049528785944722852785498719380679031698921831610135170362781179, s=23715479445748255656450369101853830474610702046088339580440862633939321505219, recId=0}, publicKey=0x15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:42,754","container":"besu3npn1-5d8ba-84559f59bf-ndbph","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":">>> Received PING  packet from peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@18.198.144.187:30303?discport=47715): Packet{type=PING, data=PingPacketData{from=Endpoint{host='Optional[18.198.144.187]', udpPort=40404, getTcpPort=30303}, to=Endpoint{host='Optional[3.66.85.127]', udpPort=31666, getTcpPort=31666}, expiration=1727964582, enrSeq=3}, hash=0xb5973fb98e7366954dd0b16c10b1b96d53ea383e5ae3dd80a7e81a847a69e997, signature=Signature{r=15969048044140049528785944722852785498719380679031698921831610135170362781179, s=23715479445748255656450369101853830474610702046088339580440862633939321505219, recId=0}, publicKey=0x15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:44,282","container":"besu3npn1-5d8ba-84559f59bf-ndbph","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":"<<< Sending ENRREQ packet to peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@besu3n1-3c094p.eks-europe.settlemint.com:30303?discport=40404): Packet{type=ENR_REQUEST, data=ENRRequestPacketData{expiration=1727964584}, hash=0x9ff82492d4fd930a68f12d372b526025ef21cf1b8c00d24a21e1ddc2258325e3, signature=Signature{r=57524016822231109233284507682747708559406206124337964739231663704328679370303, s=15185072012360381790333234176758348689549254803876827525962099980883054014112, recId=0}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:08:44,283","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"DiscoveryProtocolLogger","message":">>> Received ENRREQ packet from peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:33513): Packet{type=ENR_REQUEST, data=ENRRequestPacketData{expiration=1727964584}, hash=0x9ff82492d4fd930a68f12d372b526025ef21cf1b8c00d24a21e1ddc2258325e3, signature=Signature{r=57524016822231109233284507682747708559406206124337964739231663704328679370303, s=15185072012360381790333234176758348689549254803876827525962099980883054014112, recId=0}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
"DiscoveryProtocolLogger","message":"<<< Sending FINDN packet to peer 0x15473523c028c21a... (enode://15473523c028c21aec9441521ba9bd7eb8a2f1fb4f3eca20bcc9167bccad3211a641e845f956f4373690b1f013d6c6dc6b19d6a173a75a89946ba992c2b9376a@besu3n1-3c094p.eks-europe.settlemint.com:30303?discport=40404): Packet{type=FIND_NEIGHBORS, data=FindNeighborsPacketData{expiration=1727965036, target=0xe21cd5320b21ed4f33396cca045777d622e066fc34b77d0e2a341708206a49c87509163b0764407dffba964f99c736e358acb3750df6f0a708eb5ebdba460da4}, hash=0x719a4b8df0948253385b8927066dec21fe52bc9c34dd875f4c0d0e9b911bd2ab, signature=Signature{r=7156302234137472742515450432521136803200741859042142716223911145525463897894, s=448375494327937347011612633599943410879688082217354762138772123812804632750, recId=0}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}
besu {"timestamp":"2024-10-03T14:18:03,457","container":"besu3n1-3c094-d976ddbdc-5dh4r","level":"TRACE","thread":"vert.x-eventloop-thread-1","class":"Di
scoveryProtocolLogger","message":">>> Received FINDN packet from peer 0xcd60b2e724dbff92... (enode://cd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42@3.66.85.127:33513): Packet{type=FIND_NEIGHBORS, data=FindNeighborsPacketData{expiration=1727965143, target=0x83b9813dd68f18ed2dff5f53de9c29afe54f7eb050f7a3fcb6247cd26d491bec604052d0662cc135783b9174183067715da083d649126af6be80f09e2e493697}, hash=0xef5b1ca8861e3d9a3a91951e14b439a599f7a199c320e6d9b2ead22cc906329a, signature=Signature{r=5930734281918079639373759653370785077540892924311354728672549826080901475091, s=44395098578055939159378944789676415418871700911175744598721702818480945703250, recId=1}, publicKey=0xcd60b2e724dbff92fc93a36fb33437eec8eef2e055aa61b5e7209740d0d8da89e31c7e17139c365defac3f3a07bb89cced90f6e09535d5da48fd83cde19a4a42}","throwable":""}