anoma / namada

Rust implementation of Namada, a Proof-of-Stake L1 for interchain asset-agnostic privacy
https://namada.net
GNU General Public License v3.0
2.4k stars 949 forks source link

Running `test_node_connectivity` by itself locally fails on v0.6.1 #152

Closed james-chf closed 1 year ago

james-chf commented 2 years ago

There might be something I'm doing wrong with running the test, if that's the case I'll fix it and close this issue. This is happening on Ubuntu 22.04 with tendermint v0.34.13 as installed with scripts/install/get_tendermint.sh, as well as macOS.

The test test_node_connectivity fails when run this way at this position, line 118

https://github.com/anoma/anoma/blob/31affe1242d4c53b1fe7892aa6aa1f6a395df097/tests/src/e2e/ledger_tests.rs#L117-L118

The anomac command being run on L117 is something roughly like

anomac --base-dir /var/folders/cy/fxxsxwd56yg67d68vxj6l52h0000gn/T/.tmpQ91S0s --mode full transfer --source Bertha --target Albert --token XAN --amount 10.1 --fee-amount 0 --gas-limit 0 --fee-token XAN --ledger-address 127.0.0.1:27657

The output of this part of the test when running it is like:

Running: /opt/workspace/heliax/repos/anoma/primary/target/debug/anomac --base-dir /var/folders/cy/fxxsxwd56yg67d68vxj6l52h0000gn/T/.tmpQ91S0s --mode full transfer --source Bertha --target Albert --token XAN --amount 10.1 --fee-amount 0 --gas-limit 0 --fee-token XAN --ledger-address 127.0.0.1:27657
read: "2022-07-04T19:23:50.682326Z  INFO anoma_apps::cli::context: Chain ID: e2e-test.b38ebde1af7834374e4e4\r\n"
Sending Ctrl+C to command: /opt/workspace/heliax/repos/anoma/primary/target/debug/anomac --base-dir /var/folders/cy/fxxsxwd56yg67d68vxj6l52h0000gn/T/.tmpQ91S0s --mode full transfer --source Bertha --target Albert --token XAN --amount 10.1 --fee-amount 0 --gas-limit 0 --fee-token XAN --ledger-address 127.0.0.1:27657
write: "\u{3}"
read: ""

Command finished: /opt/workspace/heliax/repos/anoma/primary/target/debug/anomac --base-dir /var/folders/cy/fxxsxwd56yg67d68vxj6l52h0000gn/T/.tmpQ91S0s --mode full transfer --source Bertha --target Albert --token XAN --amount 10.1 --fee-amount 0 --gas-limit 0 --fee-token XAN --ledger-address 127.0.0.1:27657

(the output above is from running the test with this branch which is based off of v0.6.1 with some extra test logging)

I suspect anomac is panicking immediately and it's not being printed but I'm not certain. Running the anomac command outside of the test panics (expectedly, as the ledger isn't running), and the output minus the panic looks like the output of the test.

mbp> /opt/workspace/heliax/repos/anoma/primary/target/debug/anomac --base-dir /var/folders/cy/fxxsxwd56yg67d68vxj6l52h0000gn/T/.tmpha0rns --mode full transfer --source Bertha --target Albert --token XAN --amount 10.1 --fee-amount 0 --gas-limit 0 --fee-token XAN --ledger-address 127.0.0.1:27657

2022-07-04T19:46:58.827772Z  INFO anoma_apps::cli::context: Chain ID: e2e-test.5bb727cc355289d70dce7
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value:
   0: HTTP error
   1: error trying to connect: tcp connect error: Connection refused (os error 61)

Location:
   /Users/main/.cargo/registry/src/github.com-1ecc6299db9ec823/flex-error-0.4.4/src/tracer_impl/eyre.rs:10

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Location: /opt/workspace/heliax/repos/anoma/primary/apps/src/lib/client/rpc.rs:1392

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

To reproduce

test() { export RUST_BACKTRACE=short export ANOMA_E2E_KEEP_TEMP=true export ANOMA_E2E_DEBUG=true

export TM_LOG_LEVEL=debug
export ANOMA_LOG=debug

make checksum-wasm

cargo test \
    $1 -- \
    --test-threads=1 \
    --nocapture

}

main() { set -x test $@ set +x }

main $@

- Run `.hack/run_e2e_test.sh node_connectivity &> .hack/test-log.log`
- After test finishes, look at `.hack/test-log.log`

The test fails will output like:

Keeping temporary directory at: "/tmp/.tmpzUUnwq" Error: 0: Reached a timeout for expect type of command Needle: Transaction is valid.

Location: tests/src/e2e/setup.rs:471

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ⋮ 5 frames hidden ⋮ 6: anoma_tests::e2e::setup::AnomaCmd::exp_string::{{closure}}::h8d4906091759e68b at /opt/workspace/heliax/repos/anoma/secondary/tests/src/e2e/setup.rs:471 7: core::result::Result<T,E>::map_err::h4597c49e98a3e883 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/result.rs:855 8: anoma_tests::e2e::setup::AnomaCmd::exp_string::h302ac7cab8999402 at /opt/workspace/heliax/repos/anoma/secondary/tests/src/e2e/setup.rs:468 9: anoma_tests::e2e::ledger_tests::test_node_connectivity::hae8d29765e54edaa at /opt/workspace/heliax/repos/anoma/secondary/tests/src/e2e/ledger_tests.rs:118 10: anoma_tests::e2e::ledger_tests::test_node_connectivity::{{closure}}::h1107c30eff2f33f5 at /opt/workspace/heliax/repos/anoma/secondary/tests/src/e2e/ledger_tests.rs:72 11: core::ops::function::FnOnce::call_once::h27e240c1d3b41cfc at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248 12: core::ops::function::FnOnce::call_once::hf74e913b5e655f82 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248 13: test::__rust_begin_short_backtrace::h78c51d9ac1d1cb82 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:573 14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h995c2f3d5662d6c3 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/alloc/src/boxed.rs:1872 15: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::hc7a51c6d2df87138 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panic/unwind_safe.rs:271 16: std::panicking::try::do_call::hdea109296a4489fb at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:492 17: std::panicking::try::h28e3286bc9b290de at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:456 18: std::panic::catch_unwind::h55c83e97e6cc5ef8 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panic.rs:137 19: test::run_test_in_process::h51641549f5bece08 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:596 ⋮ 24 frames hidden ⋮

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering. Run with RUST_BACKTRACE=full to include source snippets. The application panicked (crashed). Message: assertion failed: (left == right) left: 1, right: 0: the test returned a termination value with a non-zero status code (1) which indicates a failure Location: /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:185

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ⋮ 7 frames hidden ⋮ 8: core::panicking::assert_failed_inner::hbfd352e737e6ae08 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs: 9: core::panicking::assert_failed::h78009354925287c2 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:181 10: test::assert_test_result::hbee869644a71f244 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:185 11: anoma_tests::e2e::ledger_tests::test_node_connectivity::{{closure}}::h1107c30eff2f33f5 at /opt/workspace/heliax/repos/anoma/secondary/tests/src/e2e/ledger_tests.rs:72 12: core::ops::function::FnOnce::call_once::h27e240c1d3b41cfc at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248 13: core::ops::function::FnOnce::call_once::hf74e913b5e655f82 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248 14: test::__rust_begin_short_backtrace::h78c51d9ac1d1cb82 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:573 15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h995c2f3d5662d6c3 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/alloc/src/boxed.rs:1872 16: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::hc7a51c6d2df87138 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panic/unwind_safe.rs:271 17: std::panicking::try::do_call::hdea109296a4489fb at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:492 18: std::panicking::try::h28e3286bc9b290de at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:456 19: std::panic::catch_unwind::h55c83e97e6cc5ef8 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panic.rs:137 20: test::run_test_in_process::h51641549f5bece08 at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/test/src/lib.rs:596 ⋮ 24 frames hidden ⋮

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering. Run with RUST_BACKTRACE=full to include source snippets. FAILED

failures:

failures: e2e::ledger_tests::test_node_connectivity

james-chf commented 2 years ago

I ran all e2e tests locally on a very similar branch to v0.6.1 yesterday (https://github.com/anoma/anoma/pull/1202) and while various tests failed, test_node_connectivity wasn't one of them. I'm not sure what is going on but maybe there is something happening when running multiple tests that doesn't happen when running a singular one, such that test_node_connectivity will pass?

Here is the full stdout/stderr output of running make test-e2e, with --nocapture flag added. test-e2e.txt

arafey commented 2 years ago

See if this is still relevant once we switch to the 0.37 branch of tendermint (but before we switch to the 0.37 release).

james-chf commented 1 year ago

I just tested locally with namada v0.8.1 and Tendermint based off git commit 0b58342a46, and this seems to run fine now.