Azure / azure-sdk-for-rust

This repository is for the active development of the Azure SDK for Rust. For consumers of the SDK we recommend visiting Docs.rs and looking up the docs for any of libraries in the SDK.
MIT License
714 stars 248 forks source link

[Cosmos] Initial integration tests #1917

Open analogrelay opened 1 week ago

analogrelay commented 1 week ago

Here's a first bit of infrastructure for integration tests for azure_data_cosmos, and a single test.

To run these tests:

I still need to work out how to integrate these with CI, I'll start a separate conversation to discuss that.

Also, as part of running the tests locally, I discovered some failures that were masked by the fact that we don't run --all-features when testing from the CI and these tests depend on the key_auth feature. We probably can't use --all-features (because some tests require setup) but perhaps we should have a list of features we enable during "unit testing" 🤔 .

analogrelay commented 1 week ago

After some discussion with @heaths, I'm going to change this a bit already: Instead of TestAccount::from_env(), use a macro: get_test_account!() (or similar). That macro will return Ok(()) if the environment variable isn't set, thus effectively skipping the test.

analogrelay commented 1 week ago

Ok, here's a different approach, based on some discussions in Teams.

  1. "Live" tests, which depend on some set of "real" resources (including emulators) are marked with #[cfg_attr(not(livetest), ignore], which applies the #[ignore] attribute if --cfg=livetest is not specified when rustc is building the tests.
  2. In addition to the --cfg value, we specify a Cargo profile called livetest, which ensures that the build of live tests doesn't interfere with the dev build, causing the build cache to be invalidated. Ideally, we'd set the --cfg value as part of this Cargo profile, but that feature is still unstable and requires nightly cargo
  3. To make it easier to specify all the necessary options, I've defined an alias in .cargo/config.toml. Running cargo livetest ... will be equivalent to running cargo test with --profile=livetest (to use a separate build cache) and RUSTFLAGS=--cfg=livetest (to compile-in the live tests themselves).

With this, if you run the following command: cargo test --all-features --package azure_data_cosmos --test integration_databases (to run only "non-live" tests), you get this output:

     Running tests/integration_databases.rs (target/debug/deps/integration_databases-acf8e6c1c2544f49)

running 1 test
test database_crud ... ignored

The database_crud test has been ignored, as desired.

When you run cargo livetest --all-features --package azure_data_cosmos --test integration_databases (to run all tests, including livetests), but without specifying mandatory env vars, you get this output:

     Running tests/integration_databases.rs (target/livetest/deps/integration_databases-ba888bd60bb98b5c)

running 1 test
test database_crud ... FAILED

failures:

---- database_crud stdout ----
Error: "failed to read AZURE_COSMOS_CONNECTION_STRING environment variable: environment variable not found"

failures:
    database_crud

The "live" test has correctly failed because you requested to run it but failed to provide mandatory configuration. If you then configure the AZURE_COSMOS_CONNECTION_STRING value correctly, and re-run the test, it runs as expected:

     Running tests/integration_databases.rs (target/livetest/deps/integration_databases-ba888bd60bb98b5c)

running 1 test
test database_crud ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s
analogrelay commented 2 days ago

This is on hold as we get the "official" plan for integration/live testing in place.