ci(test): use cargo nextest to speed up the test

pan93412 commented 2 years ago

Why introducing this change?

The current cargo test is so slow that can't complete the CI in a reasonable period.

The features of Nextest

Clean, beautiful user interface. Nextest presents its results concisely so you can see which tests passed and failed at a glance.
Up to 60% faster than cargo test. Nextest uses a state-of-the-art execution model for faster, more reliable test runs.
Identify slow tests. Use nextest to detect tests that take a long time to run, and identify bottlenecks during test execution.
Designed for CI. Nextest addresses real-world pain points with continuous integration scenarios.
Use pre-built binaries for quick installation.
Set up CI-specific configuration profiles. Reuse builds and partition test runs across multiple CI jobs. (Check out this example on GitHub Actions).
Automatically retry failing tests, and mark them as flaky if they pass later. Print failing output at the end of test runs.
Output information about test runs as JUnit XML.
Cross-platform. Nextest works on Linux and other Unixes, Mac and Windows, so you get the benefits of faster test runs no matter what platform you use. ... and more coming soon!

Source: https://nexte.st

Why separating `nextest` and `doctest`?

cargo nextest hasn't supported doctest.
cargo nextest seems currently failed so cargo test --doc can't be run.

soulmachine commented 2 years ago

Would you mind explaining why Nextest is faster than cargo test?

pan93412 commented 2 years ago

Would you mind explaining why Nextest is faster than cargo test?

The original cargo test timed out due to a blocking test:

https://github.com/crypto-crawler/crypto-crawler-rs/runs/7249902584?check_suite_focus=true

While nextest ran it in a reasonable time:

https://github.com/pan93412/crypto-crawler-rs/runs/7254465565?check_suite_focus=true

Sorry for forgot pasting the actions link here.

pan93412 commented 2 years ago

@soulmachine The cargo nextest run test has been done – It took about 32 minutes instead of infinitely.

soulmachine commented 2 years ago

@soulmachine The cargo nextest run test has been done – It took about 32 minutes instead of infinitely.

nextest run many test cases in parallel, I think this is why it is fast. But it caused 429 Too Many Requests, can you reduce its threads?

pan93412 commented 2 years ago

blocking: https://github.com/nextest-rs/nextest/issues/158#issuecomment-1179666291

soulmachine commented 2 years ago

blocking: nextest-rs/nextest#158 (comment)

Got it , let's wait for a few days. The goal here is to make cargo test pass.

pan93412 commented 2 years ago

No 429 issues anymore; however, some tests did not still passed.

Bitmex

        FAIL [   3.811s]     crypto-markets::bitmex fetch_all_symbols

--- STDOUT:                  crypto-markets::bitmex fetch_all_symbols ---

running 1 test
test fetch_all_symbols ... FAILED

failures:

failures:
    fetch_all_symbols

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 3.80s

--- STDERR:                  crypto-markets::bitmex fetch_all_symbols ---
thread 'fetch_all_symbols' panicked at 'assertion failed: !symbols.is_empty()', crypto-markets/tests/bitmex.rs:13:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.768s]     crypto-markets::bitmex fetch_quanto_future_symbols

--- STDOUT:                  crypto-markets::bitmex fetch_quanto_future_symbols ---

running 1 test
test fetch_quanto_future_symbols ... FAILED

failures:

failures:
    fetch_quanto_future_symbols

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.76s

--- STDERR:                  crypto-markets::bitmex fetch_quanto_future_symbols ---
thread 'fetch_quanto_future_symbols' panicked at 'assertion failed: !symbols.is_empty()', crypto-markets/tests/bitmex.rs:85:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

------------
     Summary [ 152.534s] 78 tests run: 76 passed, 2 failed, 2 skipped
error: test run failed

Others

        FAIL [   0.007s]                      crypto-message compact::message::tests::test_calculate_hash

--- STDOUT:                                   crypto-message compact::message::tests::test_calculate_hash ---

running 1 test
test compact::message::tests::test_calculate_hash ... FAILED

failures:

failures:
    compact::message::tests::test_calculate_hash

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.00s

--- STDERR:                                   crypto-message compact::message::tests::test_calculate_hash ---
thread 'compact::message::tests::test_calculate_hash' panicked at 'assertion failed: `(left == right)`
  left: `12658250145686044913`,
 right: `4200729777880827930`', crypto-message/src/compact/message.rs:527:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

     TIMEOUT [ 120.118s]               crypto-crawler::huobi test_crawl_candlestick::_markettype_spot

--- STDOUT:                            crypto-crawler::huobi test_crawl_candlestick::_markettype_spot ---

running 1 test
test test_crawl_candlestick::_markettype_spot has been running for over 60 seconds

        FAIL [   0.007s]           crypto-msg-parser::okx_v3 l2_event::option_snapshot

--- STDOUT:                        crypto-msg-parser::okx_v3 l2_event::option_snapshot ---

running 1 test
test l2_event::option_snapshot ... FAILED

failures:

failures:
    l2_event::option_snapshot

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.00s

--- STDERR:                        crypto-msg-parser::okx_v3 l2_event::option_snapshot ---
thread 'l2_event::option_snapshot' panicked at 'assertion failed: `(left == right)`
  left: `906.0`,
 right: `90.60000000000001`', crypto-msg-parser/tests/okx_v3.rs:484:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.006s]           crypto-msg-parser::okx_v3 trade::option

--- STDOUT:                        crypto-msg-parser::okx_v3 trade::option ---

running 1 test
test trade::option ... FAILED

failures:

failures:
    trade::option

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.00s

--- STDERR:                        crypto-msg-parser::okx_v3 trade::option ---
thread 'trade::option' panicked at 'assertion failed: `(left == right)`
  left: `4.0`,
 right: `0.4`', crypto-msg-parser/tests/okx_v3.rs:179:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.108s]              crypto-rest-client::zb test_spot_l2_snapshot

--- STDOUT:                           crypto-rest-client::zb test_spot_l2_snapshot ---

running 1 test
test test_spot_l2_snapshot ... FAILED

failures:

failures:
    test_spot_l2_snapshot

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.09s

--- STDERR:                           crypto-rest-client::zb test_spot_l2_snapshot ---
thread 'test_spot_l2_snapshot' panicked at 'called `Result::unwrap()` on an `Err` value: Error("Failed zb spot btc_usdt after retrying 3 times")', crypto-rest-client/tests/zb.rs:9:79
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

     TIMEOUT [ 120.111s]                  crypto-crawler::zb test_crawl_l2_snapshot::_markettype_spot_btc_usdt_

--- STDOUT:                               crypto-crawler::zb test_crawl_l2_snapshot::_markettype_spot_btc_usdt_ ---

running 1 test
test test_crawl_l2_snapshot::_markettype_spot_btc_usdt_ has been running for over 60 seconds

        SLOW [>120.000s]                  crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot
     TIMEOUT [ 120.111s]                  crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot

--- STDOUT:                               crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot ---

running 1 test
test test_crawl_l2_snapshot_without_symbol::_markettype_spot has been running for over 60 seconds

------------
     Summary [ 585.959s] 1609 tests run: 1602 passed, 4 failed, 3 timed out, 118 skipped

Seems all of them should not be related to this PR.

soulmachine commented 2 years ago

Hi @pan93412 , thank you for the hard work, we're closer! I'll help you investigate these failed test cases.

soulmachine commented 2 years ago

@pan93412 can you give me push access to https://github.com/pan93412/crypto-crawler-rs so that I can add commits to this PR?

soulmachine commented 2 years ago

@pan93412 To make all tests pass, please comment out line 59 and 71 of crypto-crawler/tests/zb.rs

crypto-crawler / crypto-crawler-rs