Fix soroban-test integration tests to make sure the full suite is being run in CI/CD

[elizabethengelman]

I noticed that when running cargo test --package soroban-test --test it locally, or in a GH Action, not all of the tests seem to run. The output looks like this:

running 50 tests
test arg_parsing::parse_bool ... ok
test arg_parsing::parse_i32 ... ok
...
test arg_parsing::parse_u32 ... ok
test arg_parsing::parse_u64 ... ok
test config::generate_key_on_testnet ... ok
test arg_parsing::parse_enum_const ... ok
test arg_parsing::parse_enum ... ok
test arg_parsing::parse_i256 ... ok
test arg_parsing::parse_obj ... ok
test config::seed_phrase ... ok
test config::read_key ... ok
Example contract method which takes a struct
Usage Notes:
Each arg has a corresponding --<arg_name>-file-path which is a path to a file containing the corresponding JSON argument.
Note: The only types which aren't JSON are Bytes and BytesN, which are raw bytes

Usage: strukt_hel [OPTIONS]

Options:
      --strukt <{ a: u32, b: bool, c: Symbol }>
          This is from the rust doc above the struct Test
          Example:
            --strukt '{ "a": 1, "b": true, "c": "hello" }'

  -h, --help
          Print help (see a summary with '-h')

The output doesn't include the final test result including the number of failed and passed tests (i.e. test result: FAILED, 31 passed; 10 failed;...), nor does it include the status of all of the config tests, or any of the build, init, plugin, or version tests.

After digging a little bit, it seems like something in the help.rs file is causing this, though I'm not 100% sure what is going on. It looks like help.rs is potentially capturing the stdout (and maybe stderr too), so that the rest of the test output doesn't make it to the command line.

I discovered this because there are a couple of tests that should be failing now, but they aren't.

Repo steps

To see this, comment out the tests in help.rs and you'll see that there are some failing tests. I'm not sure if all of them are valid failures, but the error inversion.rs looks legit.

➜  stellar-cli git:(main) cargo test --package soroban-test --test it
    Finished `test` profile [unoptimized + debuginfo] target(s) in 17.30s
     Running tests/it/main.rs (target/debug/deps/it-d93d586aa933d606)

running 41 tests
test arg_parsing::parse_bool ... ok
test arg_parsing::parse_i128 ... ok
test arg_parsing::parse_i32 ... ok
test arg_parsing::parse_null ... ok
test arg_parsing::parse_bytesn ... ok
test arg_parsing::parse_bytes ... ok
test arg_parsing::parse_bytesn_when_hex_is_all_numbers ... ok
test arg_parsing::parse_bytes_when_hex_is_all_numbers ... ok
test arg_parsing::parse_optional_bool_with_no_quotation_marks ... ok
test arg_parsing::parse_symbol ... ok
test arg_parsing::parse_symbol_with_no_quotation_marks ... ok
test arg_parsing::parse_optional_symbol_with_no_quotation_marks ... ok
test arg_parsing::parse_u128 ... ok
test arg_parsing::parse_u32 ... ok
test arg_parsing::parse_u64 ... ok
test config::generate_key_on_testnet ... ok
test arg_parsing::parse_enum_const ... ok
test arg_parsing::parse_obj ... ok
test arg_parsing::parse_enum ... ok
test arg_parsing::parse_i256 ... ok
test config::seed_phrase ... ok
test config::read_key ... ok
test init::init ... ok
test plugin::list ... FAILED
test plugin::soroban_hello ... FAILED
test build::build_no_package_found ... ok
test build::build_default_members ... ok
test build::build_all ... ok
test build::build_package_by_name ... ok
test build::build_all_when_in_non_package_directory ... FAILED
test build::build_package_by_current_dir ... ok
test config::set_and_remove_global_network ... FAILED
test config::generate_key ... ok
test version::version ... FAILED
test config::use_env ... ok
test plugin::has_no_path_failure ... ok
test config::multiple_networks ... FAILED
test config::set_and_remove_network ... FAILED
test config::use_default_futurenet ... FAILED
test config::use_default_testnet ... FAILED
test init::init_and_deploy ... FAILED

failures:
...

---- version::version stdout ----
thread 'version::version' panicked at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5:
Unexpected stdout, failed diff original var
├── original: soroban 21.5.0 (v20.0.0-329-g3f06e1dd23fd1b892dc1d1fac2cd406dee4fabf4)
│   stellar-xdr 22.0.0-rc.1.1 (72e523004b5906eb1829990f9b14d2f0fa3018f0)
│   xdr curr (529d5176f24c73eeccfa5eba481d4e89c19b1181)
├── diff:
│   ---     orig
│   +++     var
│   @@ -1 +1 @@
│   -soroban 21.5.0 (v20.0.0-329-g3f06e1dd23fd1b892dc1d1fac2cd406dee4fabf4)
│   +stellar 21.5.0 (v20.0.0-359-gfdbd9b847d73ae932afeb409f32c86d922194844-dirty)
└── var as str: stellar 21.5.0 (v20.0.0-359-gfdbd9b847d73ae932afeb409f32c86d922194844-dirty)
    stellar-xdr 22.0.0-rc.1.1 (72e523004b5906eb1829990f9b14d2f0fa3018f0)
    xdr curr (529d5176f24c73eeccfa5eba481d4e89c19b1181)

...

failures:
    build::build_all_when_in_non_package_directory
    config::multiple_networks
    config::set_and_remove_global_network
    config::set_and_remove_network
    config::use_default_futurenet
    config::use_default_testnet
    init::init_and_deploy
    plugin::list
    plugin::soroban_hello
    version::version

test result: FAILED. 31 passed; 10 failed; 0 ignored; 0 measured; 0 filtered out; finished in 50.61s

[juandiegocv27]

Hi @elizabethengelman can i work in this issue? I have experience with rust and testing👋🏻

[Ifropc]

Integration tests were not running properly, as mentioned in this issue. It was caused by help tests exiting suite early with exit code 0, therefore effectively ignoring all other tests. Help tests are fixed in #1734 and as the result failures are now visible:

failures:
    build::build_all_when_in_non_package_directory
    config::generate_key
    config::multiple_networks
    config::set_and_remove_global_network
    config::set_and_remove_network
    config::set_default_identity
    config::set_default_network
    config::use_default_futurenet
    config::use_default_testnet
    init::init
    init::init_and_deploy
    version::version

All failures are addressed and fixed in different PRs:

• build: #1736
• config + version: #1737
• init: #1738