search

scylladb / scylla-rust-driver

Async CQL driver for Rust, optimized for ScyllaDB!
Apache License 2.0
582 stars 101 forks source link

Run all examples in the CI #839

Open cvybhu opened 1 year ago

cvybhu commented 1 year ago

The CI checks that all of the code examples compile, but it doesn't check whether they actually work. The code is constantly evolving, and things that used to work in the past might stop working at some point, without anyone noticing.

In https://github.com/scylladb/scylla-rust-driver/issues/835 a user reported that one of the examples doesn't work. It fails because the specified quorum is invalid for this type of request. This should've been caught in the PR that introduced this example, but wasn't.

It would be good to run all of the examples in the CI. This will help us catch all breakages.

cvybhu commented 1 year ago

I don't know any simple way to run all of the examples using a single command. There's an issue about it: https://github.com/rust-lang/cargo/issues/8356

We can just list every example as a separate step in the workflow, there aren't that many of them.

piodul commented 1 year ago

Some of the examples cannot be just run like you would run a test - some of them require some CLI options (allocations.rs), user input (cqlsh-rs.rs) or specific environment (cloud.rs).

IMO those tests that don't need any user input should just be integration tests or maybe examples in docstrings (and cargo test runs those). Most of the existing examples could be treated in that way. Not sure about the others, but we could consider what to do with them on a per-case basis.

nsipplswezey commented 1 year ago

Since I'm the user who reported an error in one of the examples, I figured I'd open a PR https://github.com/scylladb/scylla-rust-driver/pull/845 taking a look at running examples in CI. After running the examples in CI, I agree with @piodul that the examples aren't all consistent enough to run like you would run a test. Along with the examples @piodul highlighted, I just opened this issue https://github.com/scylladb/scylla-rust-driver/issues/844 that I think is caused by two examples that use the same table naming conventions that cause the other to error when run against the same scylladb cluster. Also the tls example is already in CI. And the cloud example might need a review of its config credentials.

I think a per-case basis certainly makes sense. Hope this little bit of work is helpful.