hyperledger / besu

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client https://wiki.hyperledger.org/display/besu
https://www.hyperledger.org/projects/besu
Apache License 2.0
1.53k stars 849 forks source link

Without the --net host option, Docker Besu cannot find peers #5672

Open hardice501 opened 1 year ago

hardice501 commented 1 year ago

Description

We are Test Besu 21.10.9 Docker, IBFT2 consensus protocol, on a private network, gas price free with 4 validators.

4 Validators on 1 machine is no problem.

But build docker on each of 4 computers with different ip, bootnode cannot found each peers.

Acceptance Criteria

Steps to Reproduce (Bug)

  1. By ibftConfigFile.json, make genesis file and key file in first Computer
  2. 1 Validator(also bootnode) setup to first Computer without --net host option,
  3. Copy and paste the Genesis file and key file to the second computer
  4. Run second Computer's Docker without --net host option
  5. ping test in docker container

Expected behavior: [What you expect to happen] Successfully connected peers Successfully Ping test Actual behavior: [What actually happens] No connection between peers. Successfully Ping test Frequency: [What percentage of the time does it occur?] Always except when adding the --net host option Ping test always succeeds even without --net host option

Logs (if a bug)

Please post relevant logs from Besu (and the consensus client, if running proof of stake) from before and after the issue. FullSyncTargetManager | No sync target, waiting for peers: 0

Versions (Add all that apply)

Additional Information (Add any of the following or anything else that may be relevant)

hardice501 commented 1 year ago

update issue in my cases, add option -p 30303:30303/udp is working at linux.

but in mac(m1), must need to build a virtual environment to run the docker cli.

// the other are free license

in boot_node

> admin.peers
{
  firstEnode: {
     localhost: [virtual ip]:30303,
     remotehost: [virtual ip]:{random port},
 },
 secondEnode:{
   localhost: [virtual ip]:30303,
   remotehost: [virtual ip]:{random port},
 }
}

so each peers(ex: first and second p2p connection) cannot connect.

non-fungible-nelson commented 1 year ago

Thanks for the update - Docker on Mac has some limitations with Localhost, recommend testing/dev with --net.

Definitely also recommend updating to the latest version and trying as well. 23.4.4 Not sure if this version of Besu (21.10.9) did not have specific M1 support in Besu. A newer version might play nicer with Docker.

hardice501 commented 1 year ago

Thanks for the update - Docker on Mac has some limitations with Localhost, recommend testing/dev with --net.

Definitely also recommend updating to the latest version and trying as well. 23.4.4 Not sure if this version of Besu (21.10.9) did not have specific M1 support in Besu. A newer version might play nicer with Docker.

Thanks for the reply. A nonce error always occurred when performing a load test on besu version 21.10.9 later. Do you know anything about this?

non-fungible-nelson commented 1 year ago

We have tweaked nonce behaviors a lot to prevent Denial of Service attacks on Mainnet. Newer versions of Besu should have fixes for this and we even have a new transaction pool type that can handle "future nonce" transactions.

This flag has more details: https://besu.hyperledger.org/stable/public-networks/reference/cli/options#tx-pool-limit-by-account-percentage

hardice501 commented 1 year ago

We have tweaked nonce behaviors a lot to prevent Denial of Service attacks on Mainnet. Newer versions of Besu should have fixes for this and we even have a new transaction pool type that can handle "future nonce" transactions.

This flag has more details: https://besu.hyperledger.org/stable/public-networks/reference/cli/options#tx-pool-limit-by-account-percentage

I try that(ex tx-pool-limit-by-account-percentage=0.9) and try https://github.com/hyperledger/besu/pull/5290 this also. but nonce error always occurred.(if sended TPS is 1000). How can i solve it?

non-fungible-nelson commented 1 year ago

Have you tried setting the limit to 1 (to allow all future nonce transactions)?

tx-pool-limit-by-account-percentage=1

If you are in a private network with known senders, this should stop the nonce gap error you're seeing.

If this doesn't work. @fab-10 might have some insight on how to avoid this nonce issue.

fab-10 commented 1 year ago

@hardice501 which is your block time and block gas limit? since your TPS is quite high, you need some more tuning:

Let me know if these options solve the nonce issue, otherwise we can debug further.

hardice501 commented 1 year ago

What's mean of TPSblock-time2 ??

my besu options is below.

besu --node-private-key-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/key --rpc-http-port=8545 --rpc-ws-port=8546 --p2p-port=30303 --genesis-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/genesis.json --data-path=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/database --rpc-http-max-active-connections=1000 --rpc-http-enabled --rpc-http-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG,TRACE --rpc-http-cors-origins=all --rpc-ws-enabled --rpc-ws-host=0.0.0.0 --rpc-ws-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG --min-gas-price=0 --rpc-ws-max-frame-size=104857600 --Xlayered-tx-pool-layer-max-capacity=50000000000 --Xlayered-tx-pool-max-prioritized=160000 --Xlayered-tx-pool-max-future-by-sender=160000 --host-allowlist="*"

caliper options and genesis options: blockperiodseconds: 1 Using tool: Caliper-benchmark (websocket)

simpleArgs: &simple-args initialMoney: 10000 moneyToTransfer: 100 numberOfAccounts: &number-of-accounts 10

test: name: simple description: >- This is an example benchmark for Caliper, to test the backend DLT's performance with simple account opening & querying transactions. workers: number: 3 rounds:

java heap size : java -XX:+PrintFlagsFinal -version 2>&1 | grep -i -E 'heapsize|metaspacesize|version'
size_t ErgoHeapSizeLimit = 0 {product} {default} size_t HeapSizePerGCThread = 43620760 {product} {default} size_t InitialHeapSize = 10737418240 {product} {command line} size_t LargePageHeapSizeThreshold = 134217728 {product} {default} size_t MaxHeapSize = 17179869184 {product} {ergonomic} size_t MaxMetaspaceSize = 18446744073709551615 {product} {default} size_t MetaspaceSize = 22020096 {product} {default} size_t MinHeapSize = 10737418240 {product} {command line} uintx NonNMethodCodeHeapSize = 5839564 {pd product} {ergonomic} uintx NonProfiledCodeHeapSize = 122909338 {pd product} {ergonomic} uintx ProfiledCodeHeapSize = 122909338 {pd product} {ergonomic} size_t SoftMaxHeapSize = 17179869184 {manageable} {ergonomic}

also I already try to default txpool option.

besu --node-private-key-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/key --rpc-http-port=8545 --rpc-ws-port=8546 --p2p-port=30303 --genesis-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/genesis.json --data-path=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/database --rpc-http-max-active-connections=1000 --rpc-http-enabled --rpc-http-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG,TRACE --rpc-http-cors-origins=all --rpc-ws-enabled --rpc-ws-host=0.0.0.0 --rpc-ws-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG --min-gas-price=0 --rpc-ws-max-frame-size=10485760 --tx-pool-limit-by-account-percentage=1 --tx-pool-max-size=16000 --host-allowlist="*" --graphql-http-enabled=true --revert-reason-enabled=true

But, both of options cannot work by nonce error. only 21.10.9 version is working.

fab-10 commented 1 year ago

TPSblock-time2 is actually TPS block-time 2, markdown interpreted the *

in the first command, you are missing the --Xlayered-tx-pool=true option, so you are still using the legacy pool, try this command leaving the defaults value and see the output

besu --node-private-key-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/key --rpc-http-port=8545 --rpc-ws-port=8546 --p2p-port=30303 --genesis-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/genesis.json --data-path=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/database --rpc-http-max-active-connections=1000 --rpc-http-enabled --rpc-http-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG,TRACE --rpc-http-cors-origins=all --rpc-ws-enabled --rpc-ws-host=0.0.0.0 --rpc-ws-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG --min-gas-price=0 --rpc-ws-max-frame-size=104857600 --host-allowlist="*" --Xlayered-tx-pool=true
hardice501 commented 1 year ago

Thank u for comment.

But I have same problem in this options.(add --Xlayered-tx-pool=true) my besu version is latest. besu --node-private-key-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/key --rpc-http-port=8545 --rpc-ws-port=8546 --p2p-port=30303 --genesis-file=/Users/songsanghyeon/work/Constructor-Besu-IBFT/genesis.json --data-path=/Users/songsanghyeon/work/Constructor-Besu-IBFT/Node-1/database --rpc-http-max-active-connections=1000 --rpc-http-enabled --rpc-http-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG,TRACE --rpc-http-cors-origins=all --rpc-ws-enabled --rpc-ws-host=0.0.0.0 --rpc-ws-apis=ETH,NET,QBFT,ADMIN,PRIV,EEA,MINER,WEB3,TXPOOL,DEBUG --min-gas-price=0 --rpc-ws-max-frame-size=104857600 --Xlayered-tx-pool-layer-max-capacity=50000000000 --Xlayered-tx-pool-max-prioritized=160000 --Xlayered-tx-pool-max-future-by-sender=160000 --host-allowlist="*" --graphql-http-enabled=true --revert-reason-enabled=true --Xlayered-tx-pool=true

and when "Failed tx on simple calling method transfer nonce" error occured(some transaction is pended), Even if lower the TPS to 10, the same error continues to occur.

How to handling nonce error problem(high TPS) and pended problem(after nonce error)?

fab-10 commented 1 year ago

At this point, I think I need to try to reproduce your test locally.