search

paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.78k stars 1.02k forks source link

CD between reth_payload_builder and reth_node_api #6258

Closed DoTheBestToGetTheBest closed 7 months ago

DoTheBestToGetTheBest commented 7 months ago

Describe the bug

I've identified a cyclic dependency between 

# reth-node-api/Cargo.toml
[dev-dependencies]
# for examples
reth-payload-builder.workspace = true

# reth-payload-builder/Cargo.toml
[dependencies]
...
reth-node-api.workspace = true

We should eliminate cd. We could refactor code, moving functionality between crates, or introducing new crates to break the cycles

I'm unable to run reth on wsl2 and unable to use reth crate. Also rust analyzer keep warning so it's impossible to do anything

Steps to reproduce

having wsl2 installed and rust analyzer

Node logs

no log

Platform(s)

Windows (x86)

What version/commit are you on?

main

What database version are you on?

none

What type of node are you running?

Archive (default)

What prune config do you use, if any?

none

If you've built Reth from source, provide the full command you used

From source.

Code of Conduct

Rjected commented 7 months ago

Do you mind sharing which command is failing and the logs from that? The cyclic dependencies are there only for examples, which compile fine as far as I remember

DoTheBestToGetTheBest commented 7 months ago

Do you mind sharing which command is failing and the logs from that? The cyclic dependencies are there only for examples, which compile fine as far as I remember

Sure

[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))
[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))
[ERROR rust_analyzer::main_loop] FetchBuildDataError:
error: failed to run custom build command for `libffi-sys v2.3.0`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `reth/target/debug/build/libffi-sys-ddce524bb04d37cd/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  checking build system type... x86_64-pc-linux-gnu
  checking host system type... x86_64-pc-linux-gnu
  checking target system type... x86_64-pc-linux-gnu
  continue configure in default builddir "./x86_64-unknown-linux-gnu"
  ....exec /bin/bash .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux
  gnu"

  --- stderr
  .././configure: line 2247: config.log: No such file or directory
  .././configure: line 2257: config.log: No such file or directory
  cat: standard output: No such file or directory
  thread 'main' panicked at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libffi-sys-2.3.0/build/common.rs:8:5:
  Configuring libffi
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/62d7ed4a6775c4490e493093ca98ef7c215b835b/library/std/src/panicking.rs:647:5
     1: core::panicking::panic_fmt
               at /rustc/62d7ed4a6775c4490e493093ca98ef7c215b835b/library/core/src/panicking.rs:72:14
     2: core::panicking::panic_display
     3: build_script_build::common::run_command
     4: build_script_build::not_msvc::configure_libffi
     5: build_script_build::not_msvc::build_and_link
     6: build_script_build::main
     7: core::ops::function::FnOnce::call_once
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))
Rjected commented 7 months ago

Do you mind sharing which command is failing and the logs from that? The cyclic dependencies are there only for examples, which compile fine as far as I remember

Sure

[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))
[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))
[ERROR rust_analyzer::main_loop] FetchBuildDataError:
error: failed to run custom build command for `libffi-sys v2.3.0`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `reth/target/debug/build/libffi-sys-ddce524bb04d37cd/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  checking build system type... x86_64-pc-linux-gnu
  checking host system type... x86_64-pc-linux-gnu
  checking target system type... x86_64-pc-linux-gnu
  continue configure in default builddir "./x86_64-unknown-linux-gnu"
  ....exec /bin/bash .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux
  gnu"

  --- stderr
  .././configure: line 2247: config.log: No such file or directory
  .././configure: line 2257: config.log: No such file or directory
  cat: standard output: No such file or directory
  thread 'main' panicked at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libffi-sys-2.3.0/build/common.rs:8:5:
  Configuring libffi
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/62d7ed4a6775c4490e493093ca98ef7c215b835b/library/std/src/panicking.rs:647:5
     1: core::panicking::panic_fmt
               at /rustc/62d7ed4a6775c4490e493093ca98ef7c215b835b/library/core/src/panicking.rs:72:14
     2: core::panicking::panic_display
     3: build_script_build::common::run_command
     4: build_script_build::not_msvc::configure_libffi
     5: build_script_build::not_msvc::build_and_link
     6: build_script_build::main
     7: core::ops::function::FnOnce::call_once
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[ERROR project_model::workspace] cyclic deps: reth_auto_seal_consensus(Idx::<CrateData>(516)) -> reth(Idx::<CrateData>(513)), alternative path: reth(Idx::<CrateData>(513)) -> reth_auto_seal_consensus(Idx::<CrateData>(516))
[ERROR project_model::workspace] cyclic deps: reth_network(Idx::<CrateData>(558)) -> reth_network(Idx::<CrateData>(558)), alternative path: reth_network(Idx::<CrateData>(558))
[ERROR project_model::workspace] cyclic deps: reth_payload_builder(Idx::<CrateData>(568)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568))
[ERROR project_model::workspace] cyclic deps: reth_provider(Idx::<CrateData>(574)) -> reth_node_api(Idx::<CrateData>(563)), alternative path: reth_node_api(Idx::<CrateData>(563)) -> reth_payload_builder(Idx::<CrateData>(568)) -> reth_provider(Idx::<CrateData>(574))
[ERROR project_model::workspace] cyclic deps: reth_stages(Idx::<CrateData>(588)) -> reth_blockchain_tree(Idx::<CrateData>(521)), alternative path: reth_blockchain_tree(Idx::<CrateData>(521)) -> reth_stages(Idx::<CrateData>(588))
[ERROR project_model::workspace] cyclic deps: reth_trie(Idx::<CrateData>(598)) -> reth_provider(Idx::<CrateData>(574)), alternative path: reth_provider(Idx::<CrateData>(574)) -> reth_trie(Idx::<CrateData>(598))

what's the actual cargo command used to get the output here? Is this from an editor / IDE?

DoTheBestToGetTheBest commented 7 months ago

what's the actual cargo command used to get the output here? Is this from an editor / IDE?

as i said, i don't run any command cargo because rust analyzer is blocked by this so i can't use anything related to reth

DoTheBestToGetTheBest commented 7 months ago

@mattsse hey what do you suggest for this ?

Rjected commented 7 months ago

Looks like this is what is happening https://github.com/rust-lang/rust-analyzer/issues/14167

agree with one point from the comments:

Being able to have cycles in dev deps is an important bit of functionality for integration tests, doctests, examples, and benches. It does not have strange behavior there.

Pretty sure we would not be able to have examples in the node-api crate without a cyclic dep, regardless of how we shuffle crates around

DoTheBestToGetTheBest commented 7 months ago

Pretty sure we would not be able to have examples in the node-api crate without a cyclic dep, regardless of how we shuffle crates around

Thank you for your answer Rjected, i'm curious to know what could be the solution for this problem now ?

Do you have any idea?

DaniPopes commented 7 months ago

Is this an actual problem? Like what can you notice that doesn't work, where breaking the cycle would fix it? AFAIK this isn't an actual issue, since dev-dependencies are allowed to be circular

mattsse commented 7 months ago

yea I think @DoTheBestToGetTheBest only has these issues in RA, not familiar with that IDE so unclear if this is solvable with settings

DaniPopes commented 7 months ago

I do too, but I can't really notice anything

mattsse commented 7 months ago

in anycase this will be addressed by #6333

Degen-Player commented 7 months ago

Is this an actual problem? Like what can you notice that doesn't work, where breaking the cycle would fix it? AFAIK this isn't an actual issue, since dev-dependencies are allowed to be circular

I have same problem as OP https://github.com/paradigmxyz/reth/issues/6128

Rust analyzer is probably the tool used by 70% of rust user it's not an IDE like said here

DoTheBestToGetTheBest commented 7 months ago

in anycase this will be addressed by #6333

Thank you guys for this fix !! Amzing from the part of reth team!

Blaeaea commented 7 months ago

Same problem with rust analyzer can't use reth anymore

Blaeaea commented 7 months ago

Not in WSL2 for my part in in macos