(discussion): Help w/ rbuilder (taiko-gwyneth)

adaki2004 commented 2 months ago

Hey Guys,

First of all thanks if you checking it out ! :)

About our current setup! We want to setup L2 network(s) for testing at Taiko's Gwyneth initiative. (Yes, multiple, so called 'booster rollups' for horizontal scaling.) We want rbuilder be shot up for testing, currently we are under heavy dev. phase !

We have a reth based network. Shoot up with kurtosis.

Very high level TLDR about our current design (and testing E2E) be like:

Submit transactions to our L2 nodes
builders (rbuilder) would listen, then after X txs (or being profitable, whatever) it would proposeBlock() towards an L1 smart contract
our node would index those smart contract events, and then derive the txList (from calldata or blob), then decode and execute them actually on L2

What i did:

I incorporated the rbuilder binary into the EL image so that i can run the rbuilder run config-gwyneth-reth.toml

The config-gwyneth-reth.toml is the following:

The config file i'm using is the following:
log_json = false
log_level = "info,rbuilder=debug"
telemetry_port = 6060
telemetry_ip = "0.0.0.0"

chain = "/network-configs/genesis.json"
reth_datadir = "/data/reth/execution-data"
relay_secret_key = "5eae315483f028b5cdd5d1090ff0c7618b18737ea9bf3c35047189db22835c48"
coinbase_secret_key = "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"

cl_node_url = ["http://172.16.140.12:4000"]
jsonrpc_server_port = 8646
jsonrpc_server_ip = "0.0.0.0"
el_node_ipc_path = "/tmp/reth.ipc-aBON5snG"
extra_data = "⚡🤖"

dry_run = false
dry_run_validation_url = "http://localhost:8545"

# blocks_processor_url can be an API service to record bids and transactions. It is not required.
# blocks_processor_url = "http://block_processor.internal"

ignore_cancellable_orders = true

sbundle_mergeabe_signers = []
# slot_delta_to_start_submits_ms is usually negative since we start bidding BEFORE the slot start
# slot_delta_to_start_submits_ms = -5000
live_builders = ["mp-ordering"]

[[relays]]
name = "custom"
url = "http://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@localhost:5555"
priority = 0
use_ssz_for_submit = false
use_gzip_for_submit = false

[[builders]]
name = "mgp-ordering"
algo = "ordering-builder"
discard_txs = true
sorting = "mev-gas-price"
failed_order_retries = 1
drop_failed_orders = true

[[builders]]
name = "mp-ordering"
algo = "ordering-builder"
discard_txs = true
sorting = "max-profit"
failed_order_retries = 1
drop_failed_orders = true

The logs i getting is the following:

# rbuilder run rbuilder/config-gwyneth-reth.toml
thread 'tokio-runtime-worker' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/warp-0.3.7/src/server.rs:217:27:
error binding to 0.0.0.0:6060: error creating server listener: Address already in use (os error 98)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-09-04T07:57:38.488789Z  INFO rbuilder::live_builder::config: Builder mev boost normal relay pubkey: 0xa1885d66bef164889a2e35845c3b626545d7b0e513efe335e97c3a45e534013fa3bc38c3b7e6143695aecc4872ac52c4
2024-09-04T07:57:38.488971Z  INFO rbuilder::live_builder::config: Builder mev boost optimistic relay pubkey: 0xa1885d66bef164889a2e35845c3b626545d7b0e513efe335e97c3a45e534013fa3bc38c3b7e6143695aecc4872ac52c4
2024-09-04T07:57:38.489000Z  INFO rbuilder::live_builder::config: Optimistic mode, enabled: false, prevalidate: false, max_value: 0.000000000000000000
2024-09-04T07:57:38.522979Z  INFO rbuilder::live_builder: Builder block list size: 0
2024-09-04T07:57:38.523002Z  INFO rbuilder::live_builder: Builder coinbase address: 0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266
2024-09-04T07:57:38.526058Z  WARN rbuilder::live_builder::order_input: ignore_cancellable_orders is set to true, some order input is ignored
2024-09-04T07:57:38.526112Z  INFO rbuilder::live_builder::payload_events: MevBoostSlotDataGenerator: started
2024-09-04T07:57:38.526206Z  INFO rbuilder::live_builder::payload_events::payload_source: PayloadSourceReconnector connecting
2024-09-04T07:57:38.544040Z  INFO rbuilder::live_builder::order_input::clean_orderpool: Clean orderpool job: started
2024-09-04T07:57:38.544343Z  INFO rbuilder::live_builder::order_input::rpc_server: RPC server job: started
2024-09-04T07:57:38.544593Z  INFO rbuilder::live_builder::order_input::txpool_fetcher: Subscribe to txpool with blobs: started
2024-09-04T07:57:38.544645Z  INFO rbuilder::live_builder::order_input: OrderPoolJobs: started
2024-09-04T07:57:42.005018Z  WARN relay{relay="custom" slot=83}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:45.501326Z  WARN relay{relay="custom" slot=83}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:46.002620Z  WARN relay{relay="custom" slot=84}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:46.049218Z  WARN relay{relay="custom" slot=84}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:54.001041Z  WARN relay{relay="custom" slot=84}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:57.501470Z  WARN relay{relay="custom" slot=84}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:58.005348Z  WARN relay{relay="custom" slot=85}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:57:58.045842Z  WARN relay{relay="custom" slot=85}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:06.003986Z  WARN relay{relay="custom" slot=85}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:09.501946Z  WARN relay{relay="custom" slot=85}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:10.007321Z  WARN relay{relay="custom" slot=86}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:10.052874Z  WARN relay{relay="custom" slot=86}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:18.001480Z  WARN relay{relay="custom" slot=86}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:21.501648Z  WARN relay{relay="custom" slot=86}: rbuilder::live_builder::payload_events::relay_epoch_cache: Relay returned error while getting epoch data, error err=RequestError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(5555), path: "/relay/v1/builder/validators", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })) })
2024-09-04T07:58:45.502708Z ERROR rbuilder::live_builder::payload_events::payload_source: Too long waiting for a Payload, reconnecting
2024-09-04T07:58:45.502746Z  INFO rbuilder::live_builder::payload_events::payload_source: PayloadSourceReconnector waiting to reconnect
2024-09-04T07:58:57.504577Z  INFO rbuilder::live_builder::payload_events::payload_source: PayloadSourceReconnector connecting
2024-09-04T07:59:21.505937Z ERROR rbuilder::live_builder::payload_events::payload_source: Too long waiting for a Payload, reconnecting
2024-09-04T07:59:21.506066Z  INFO rbuilder::live_builder::payload_events::payload_source: PayloadSourceReconnector waiting to reconnect
2024-09-04T07:59:22.004915Z ERROR rbuilder::live_builder::payload_events::payload_source: Error while sending payload event,CLPayloadSource closed
2024-09-04T07:59:33.506982Z  INFO rbuilder::live_builder::payload_events::payload_source: PayloadSourceReconnector connecting
2024-09-04T07:59:34.004467Z ERROR rbuilder::live_builder::payload_events::payload_source: Error while sending payload event,CLPayloadSource closed
...
2024-09-04T08:00:38.545019Z ERROR rbuilder::live_builder::watchdog: Watchdog timeout

Questions/observations:

I'm sure the chain is correct.
cl_node_url i suppose is correct, but i'm not 100% sure. I guess it would throw an error if not (?). The IP is correct for sure, the port i suppose from the by-default open ports.
reth_datadir i'm almost pretty sure it is not correct, because i guess i'd need to use the L2 DB (but given our current architecture even tho i create and explicitly set a static path for our L2 node DB, it seems nothing is put in there. cc @Brechtpd see: https://github.com/taikoxyz/taiko-reth/pull/39/commits/03af4b13ca4bc5907231a70a9337d5d45e8fdecb ). But can it cause the above log ?
jsonrpc_server_port: I guess it is for the rbuilder, and not referring to the el node's 8545 rpc server ?
el_node_ipc_path pretty sure it is correct.
Can you point out some flaws in config or design issues i might be missing ?

Thanks you in advance !

ferranbt commented 2 months ago

Hey! Do you guys have a relayer in the design? The current rbuilder sends the finished blocks/bids to a relayer to be included. If you want to send the block to a smart contract you have to modify the rbuilder and write an implementation of the BlockBuilderSink to decide what to do with the block.

adaki2004 commented 2 months ago

Hey! Do you guys have a relayer in the design?

Hey thanks for checking in! Yeah, i forgot to mention this is under testing only. For later usage, we need no relayer, but to "send to L1", which is not (yet) implemented, but we have a fork of rbuilder in taiko's orga, which eventually will ! :)

So here is the fork PR: https://github.com/taikoxyz/rbuilder/pulls (curently really doing just nothing at this point)

And actually what i would wanted to see is a log printed out when i send the L2 rpc a tx: https://github.com/taikoxyz/rbuilder/blob/c47792c7ffd78847cac912c15b15bba5b65884c3/crates/rbuilder/src/building/order_commit.rs#L393

ferranbt commented 2 months ago

Hey! Happy to help you with testing the integration with Taiko. Do you have any specific question?

adaki2004 commented 2 months ago

Hey! Happy to help you with testing the integration with Taiko. Do you have any specific question?

Hey Ferran!

Thank you very much for your support!

So in order to reproduce the things you need:

2 main tools:
- docker (we use docker desktop for easier debugging)
- kurtosis (recommended by reth book)

To install Kurtosis, here is a guide: https://docs.kurtosis.com/install/

Clone Taiko's reth, then switch to the proper branch: rbuilder ) git clone https://github.com/taikoxyz/taiko-reth git checkout rbuilder
After you have the repo, go into /packages/protocol and: make install -> This will build taiko's reth and also the Docker image (taiko_reth with rbuilder installed as binary + the rbuilder repo with our small, debug print modification) and It will shoot up the reth based L1 network with L2 nodes as execution extensions. (Not only the binary but the rbudiler's repo with the attached rbuilder config file will be located in/app/rbuilder directory. The config will be slightly different because i need to adapt to the ipc file (which for L2 is always some temp name) and the CL node IP wont be localhost obviously). This command will not try to run the rbuilder by default!

Afterwards, we have a "base" setup. Then what we want to see is:

rbuilder attached to that Taiko L2. (Unfortunately we still see some issues with the L2 exex node DB creation, so that is something to be solved too. cc @Brechtpd )
And when we send an L2 transaction (to L2 rpc) we would like to see the debug print println!("Dani debug: Apply tx to state!"); in commit_tx()(from where we might be able to take over :)

I suspect we have issues with the DB (since nothing wil be put into that static /app/exex_datadir folder, that is 1 suspicious thing - but i guess that is on us, and hopefully not blocking the integration.

So our TLD setup/flow is this:

User send TXs to L2-rpc (the L2 local (inside container) port is: 10110, and it is exposed to your local host instance with some random port). At this point our L2 node does not execute anything - we outsource building to rbuilder.
rbuilder is listening and batching TXs and at some point it will submit to an L1 contract (mods are on us obviously).
L2 exex nodes is listening for specific events and when L1 BlockProposed event happening, it will fetch the TX list and executes on L2.

That is it in the nutshell. Please let me know how i can serve you best, I'm also okay jumping into a call if that is better. I'm in CET timezone.

Thanks a zillion for dealing with this!

adaki2004 commented 2 months ago

@ferranbt btw to locate the L2 exex DB correct location you can always use: find / -type d -name "db"

Let me know if you would prefer a joint debug session!

adaki2004 commented 2 months ago

@ferranbt Good news, it seems it is running - tho i'm still unsure why i cannot see my debug message in commit_tx() when i send a txn to the L2 rpc: https://github.com/taikoxyz/taiko-reth/pull/39#issuecomment-2333473498

ferranbt commented 1 month ago

Hi @adaki2004. But what is the specific issue with the rbuilder, is it not creating blocks? Do you need help figuring out how to do the mods?

adaki2004 commented 1 month ago

Hi @adaki2004. But what is the specific issue with the rbuilder, is it not creating blocks? Do you need help figuring out how to do the mods?

Hey Ferran.

Yes, currently struggling to catch the debug print in commit_tx(). (Which would be the first cornerstone of creating blocks indeed).

So i suppose we subscribe to pending transactions in the EL mempool via IPC. (subscribe_to_txpool_with_blobs).

And what i guessed is (maybe wrong) that when i submit a tx to the L2 exex RPC my debug print should be triggered in commit_tx() (https://github.com/taikoxyz/rbuilder/blob/0e069a7ec68e3ad450c9a13e1174c22a42d59500/crates/rbuilder/src/building/order_commit.rs#L393). But it does not.

Tho, when i advance the chain (with a txn) the clean_orderpool subscription logs are printed out, so i somewhat assume IPC connection is good.. Then maybe i dont really get where to catch these individual transactions.

UPDATE: IPC socket connection works as expected, tested that with a stupid simple python script.

adaki2004 commented 1 month ago

@ferranbt So i've reached the point where the rbuilder is "running" and the tx (not bundle, just tx) arrives and caught properly in the route of: subscribe_to_txpool_with_blobs -> get_tx_with_blobs -> send command -> receiver intercepts it and forwards it to OrderPools process_commands() -> process_command() -> process_order() etc.

But:

Command target block: None
Sink count change from 0 to 0

According to my debug prints.

Could you please tell me if i'm doing something wrong ? Maybe the configuration is wrong ? Or this is expected with singe TX and not bundle ?
When/how the block_building (hence applyting tx to DB, commit_tx()) shall be happening and most probably what im doing wrong here ?

All i would like to do (for now) is to have a "stupid simple" block builder which would just build a block with a txn. (Obv. make it more sophisticated later.)

Thank you for your answer.

flashbots / rbuilder

(discussion): Help w/ rbuilder (taiko-gwyneth) #167