Error Executing Benchmarks

[danforbes]

I am attempting to execute benchmarks for my node, but I keep encountering the following error, which I am not able to decipher. I have tests for my benchmarks, which run just fine.

node: ./target/release/bca-node benchmark \
    --chain dev --execution=wasm --wasm-execution='compiled' \
    --pallet pallet_bca --extrinsic '*' --steps 50 --repeat 20 --output .
2021-08-19 07:54:15 panicked at 'assertion failed: `(left == right)`
  left: `<wasm:stripped>`,
 right: `<wasm:stripped>`', /home/werq/Code/bca/node/pallets/bca/src/benchmarking.rs:13:5    
Error: Input("Error executing and verifying runtime benchmark: Other(\"Wasm execution trapped: wasm trap: unreachable
    wasm backtrace:
    0: 0x6c6cd - <unknown>!rust_begin_unwind\
    1: 0x561ca - <unknown>!core::panicking::panic_fmt::hec165b0930ecb32e
    2: 0x55d0a - <unknown>!core::panicking::assert_failed_inner::h2777f1fe566ab30a
    3: 0x55f4e - <unknown>!core::panicking::assert_failed::h801fb02f69512783
    4: 0x69d80 - <unknown>!pallet_bca::benchmarking::assert_last_event::haea0c9ff657566cf
    5: 0x6a444 - <unknown>!core::ops::function::FnOnce::call_once{{vtable.shim}}::h2153dcbafd6c0fbd
    6: 0x2b40c - <unknown>!<bca_runtime::Runtime as
        frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark
        <sp_runtime::generic::block::Block
        <sp_runtime::generic::header::Header
        <u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic
        <sp_runtime::multiaddress::MultiAddress
        <<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as
        sp_runtime::traits::IdentifyAccount>::AccountId,()>,
        bca_runtime::Call,sp_runtime::MultiSignature,
        (frame_system::extensions::check_spec_version::CheckSpecVersion<bca_runtime::Runtime>,
        frame_system::extensions::check_tx_version::CheckTxVersion<bca_runtime::Runtime>,
        frame_system::extensions::check_genesis::CheckGenesis<bca_runtime::Runtime>,
        frame_system::extensions::check_mortality::CheckMortality<bca_runtime::Runtime>,
        frame_system::extensions::check_nonce::CheckNonce<bca_runtime::Runtime>,
        frame_system::extensions::check_weight::CheckWeight<bca_runtime::Runtime>,
        pallet_transaction_payment::ChargeTransactionPayment
        <bca_runtime::Runtime>)>>>>::dispatch_benchmark::ha8fe410ec22fa7bb
    7: 0x670e9 - <unknown>!Benchmark_dispatch_benchmark
    note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information\\n\")")

[gui1117]

the benchmark seems to panic due to an assertion failed. at this line /home/werq/Code/bca/node/pallets/bca/src/benchmarking.rs:13:5

I expect to be an assertion in the benchmark or initialialisation or verify block

[danforbes]

Thank you for your response, @thiolliere! Yes, there is an assert at that line (asserting last event), however I don't understand why this would work in the tests, but not the benchmarks... https://github.com/BlockchainArt/bca-substrate-node/blob/bca/pallets/bca/src/benchmarking.rs

[danforbes]

Is it possible that this benchmarking error is related to this compile-time error that I keep running into? Note, this error is resolved by running cargo clean, but it also seems suspiciously familiar to the benchmarking error that I'm encountering...

node: cargo test --features runtime-benchmarks
   Compiling bca-node v3.0.0 (/home/werq/Code/bca/node/node)
   Compiling bca-runtime v3.0.0 (/home/werq/Code/bca/node/runtime)
   Compiling pallet-bca v1.0.0 (/home/werq/Code/bca/node/pallets/bca)
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(4085ea8e8e53b1b-d4acee38df76c8c5)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p bca_node` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(4085ea8e8e53b1b-d4acee38df76c8c5): Ok(EvaluatedToOkModuloRegions)', /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/compiler/rustc_query_system/src/query/plumbing.rs:620:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.54.0 (a178d0322 2021-07-26) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation
  `sc_service::Arc<sp_blockchain::HeaderMetadataCache<sp_runtime::generic::Block<sp_runtime::generic::Header
  <u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>>: std::marker::Sync`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd
  { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs:
  [sc_transaction_pool::FullChainApi<sc_service::client::client::Client<sc_client_db::Backend
  <sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>,
  sp_runtime::OpaqueExtrinsic>>, sc_service::LocalCallExecutor<sp_runtime::generic::Block\ 
  <sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>,
  sc_client_db::Backend<sp_runtime::generic::Block<sp_runtime::generic::Header
  <u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>, sc_executor::NativeExecutor
  <service::Executor>>, sp_runtime::generic::Block<sp_runtime::generic::Header
  <u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>, bca_runtime::RuntimeApi>,
  sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>,
  sp_runtime::OpaqueExtrinsic>>], item_def_id: DefId(511:439 ~
  sc_transaction_pool[c50a]::graph::pool::ChainApi::Block) } } }`
end of query stack
error: aborting due to previous error

[gui1117]

no compile error due to incremental compilation happens regularly, I thought latest version of rustc kind of solved most of it, anyway it shouldn't be related.

[gui1117]

do you run the test with the feature runtime-benchmarks ? otherwise the module is simply ignored so the test aren't run.

Also the test run with the mocked runtime, not the chain one, it can be that the logic differ between the 2

[danforbes]

Thank you, @thiolliere - the chain specification that was being used to execute the benchmarks specified state that conflicted with the assertions in my benchmarks. I've resolved the problem 🎉