mehcode / config-rs

⚙️ Layered configuration system for Rust applications (with strong support for 12-factor applications).
Apache License 2.0
2.43k stars 206 forks source link

[release-0.13.x] chore(Cargo.toml): Better document direct deps #495

Closed polarathene closed 7 months ago

polarathene commented 7 months ago

These are improvements from an extensive investigation I recently did to simplify maintenance of lockfile generation for the CI to benefit from, while respecting the MSRV.

Commit message:

Click to view - Slight adjustments to the version fields for compatibility with `cargo +nightly update -Z direct-minimal-versions` with the MSRV of `1.56.0`. - Add `rust-version` field for leveraging `cargo +nightly update -Z msrv-policy` to generate a lockfile that respects the MSRV, and the benefit of downstreams. - Better communicate why `dev-dependencies` are required (_identify associated examples / tests_). - Avoid repeating deps again in in `dev-dependencies`. - Raise fixed `warp` dev dep to an MSRV compatible version (_for a common `tokio-util` implicit dep version_). Simplifies CI lock maintenance.
polarathene commented 7 months ago

More info

My verbosity is an issue for some, I've split this off and tried to condense / organize it. It's purely for reference to support and document the changes in the PR should anyone need more context.

Producing a compatible Cargo.lock for the MSRV

# Resolve a `Cargo.lock` with the latest compatible deps for `rust-version` (1.56.0):
cargo +nightly update -Z msrv-policy

# Minor corrections due to deps lack of proper `-Z msrv-policy` support for the MSRV
cargo update --package memchr --precise 2.5.0
cargo update --package ordered-multimap --precise 0.4.0
cargo update --package tokio-util --precise 0.7.8
Validating minimum versions build for MSRV (with 1 less pin) ``` # For verifying minimum versions of direct deps build for the MSRV is supported: # `tokio ="1.27"` (or lower) implicitly prevents needing to pin tokio-util to 0.7.8 (as newer releases require a newer tokio) cargo +nightly update -Z direct-minimal-versions -Z msrv-policy cargo update --package memchr --precise 2.5.0 cargo update --package ordered-multimap --precise 0.4.0 ```

The following passes successfully with a Cargo.lock generated from the above:

cargo +1.56.1 check --tests
cargo +1.56.1 clippy --locked --offline --all-targets --all-features -- -D warnings

Supplementary info on -Z msrv-policy and pinned versions

Click to view `-Z msrv-policy` only works on nightly, but ideally resolves within the deps semver range a version that was last known as compatible with `rust-version` in `Cargo.toml`. - This differs from the default resolver which will attempt to satisfy the latest version in semver regardless of `rust-version` compatibility. - This also differs for older toolchains that don't understand newer `Cargo.toml` syntax like introduced in Rust `1.60.0`. - Those would ignore such published crates and resolve to an earlier version of the crate implicitly. These pins are the same versions that would be resolved regardless for the MSRV. - [`memchr 2.6.0`](https://github.com/BurntSushi/memchr/blob/c4bd85f58b763080fff52fbe67081decb2899b33/Cargo.toml#L40) is an implicit dep to quite a few crates in `Cargo.lock`, including the [explicit `nom 7` dep](https://github.com/rust-bakery/nom/blob/9678b061d3ed0a6b8e31ce569e024ec25f3dd354/Cargo.toml#L39). `memchr` only adopted [`rust-version` (1.60) since the `2.6.1` release](https://crates.io/crates/memchr/versions), hence `-Z msrv-policy` is ineffective at resolving without a pin, while the MSRV toolchain resolver would resolve a compatible version as anything newer is considered invalid. - `ordered-multimap` from `rust-ini` has the same problem, except it's history is [more complicated and documented in detail in this comment](https://github.com/mehcode/config-rs/pull/492#issuecomment-1793319804) (_[`ahash` is the issue](https://github.com/tkaitchuck/aHash/pull/181#discussion_r1377129069)_). - `tokio-util` is probably better pinned, that is easier by raising the explicitly pinned `warp` patch version in `Cargo.toml`. [This is also a bit complicated](https://github.com/mehcode/config-rs/pull/492#issuecomment-1793319968), but is an example of when `rust-version` is not correctly maintained.
Additional context on deps **Minimum version:** - `async-trait`: Minimum of `0.1.2` required to support `examples/async_source`. **MSRV bumps:** - `toml`: Newer releases after `0.5.x` series require MSRV 1.60 to 1.66, while the latest `0.8.6` release requires MSRV 1.67. - `ron`: MSRV raised to 1.64 with `0.8.1`. - `rust-ini`: MSRV raised to 1.64 with `0.19.0`, while `0.20.0` (latest) raises again to MSRV 1.71.1. Both due to `ordered-multimap`. - `indexmap`: Next major `2.0.0` raises MSRV to 1.63. - `notify`: The next major (`5.0.0`) keeps the 1.56 MSRV, while the latest `6.0.0` major raises that to MSRV 1.60 - `temp-env`: Newer minor release adopts `rust-version` since `0.3.4` (latest `0.3.6` with MSRV 1.62.1).

Without -Z msrv-policy

The following will also support building without generating Cargo.lock with -Z msrv-policy. The versions need to be pinned (in either Cargo.toml or Cargo.lock).

Click to view ```toml # These are the dev-dependencies with versions that last supported MSRV `1.56` # Direct deps tokio = { version = "1.29.1", features = ["rt-multi-thread", "macros"] } # latest 1.33 requires MSRV 1.63 chrono = { version = "0.4.26", features = ["serde"] } # newer versions raise MSRV to 1.57 reqwest = "0.11.15" # newer release bumps MSRV to 1.57 (latest 0.11.22 with MSRV 1.63) # Implicit deps: scoped-tls = "1.0.0" # 1.0.1 (latest) adds rust-version 1.59 byteorder = "1.4.3" # 1.5.0 (latest) adds rust-version 1.60 log = "0.4.18" # newer versions add rust-version 1.60 (latest 0.4.20) tempfile = "3.6.0" # newer versions raise to MSRV 1.63 (latest 3.8.1) h2 = "0.3.20" # newer versions raise to MSRV 1.63 (latest 0.3.21) tokio-util = "0.7.8" # newer versions incorrectly claim MSRV 1.56 (latest 0.7.10 with MSRV 1.63) ``` Alternatively pinning via `Cargo.toml` could avoid `=` pinning and instead rely on using `-Z direct-minimal-versions`. This requires extra pins due to the nightly toolchain resolver. ```toml # Required only for nightly toolchain (eg: when using `-Z direct-minimal-versions`) # For nom: memchr = "2.5.0" # newer versions raise MSRV, but only set rust-version at 2.6.1, (latest 2.6.4 with MSRV 1.61) # For rust-ini: ordered-multimap = "0.4.0" # newer versions raise MSRV to 1.60 (latest 0.4.3 -> ahash 0.7.7) # For json5: # 2.5.7 for both pest deps is the last version supporting MSRV 1.56 (latest 2.75 with MSRV 1.61) # (Avoidable with `-Z msrv-policy) pest = "2.0.2" pest_meta = "2.0.3" ``` The MSRV toolchain will resolve `memchr` and `ordered-multimap` correctly, thus they don't need to be pinned. - Obviously the above isn't any better, but is shared to contrast against the `-Z msrv-policy` approach. - 👎 Implicit deps (_even when only for `dev-dependencies` / CI usage_) should avoid being in `Cargo.toml`. - I've documented the latest MSRV for the semver range of each dep, which better illustrates the motivation of [this recent PR](https://github.com/mehcode/config-rs/pull/487#issuecomment-1793326893) (_raising MSRV to `1.63` to keep the CI happy_).

CI uses MSRV 1.59, while this PR supports MSRV 1.56

Click to view Note that presently the CI is testing for an MSRV of `1.59`, while the README (for this branch) presently documents `1.56` as the MSRV. - It was presumably bumped in the past to make the CI pass (_seems to be a common pattern for `config-rs`?_) without resolving a compatible lockfile (_[discussed recently](https://github.com/mehcode/config-rs/pull/483#issuecomment-1775531230) as how MSRV bumps are being determined_). - This PR enables maintaining the original MSRV of `1.56.0` (_or if preferred newer_). - `rust-version` is set to communicate the actual minimum that `config-rs` will build with. - `hashbrown 0.14.2` has done similar. Their `rust-version` is only supported if you additionally [pin a dep](https://github.com/rust-lang/hashbrown/blob/778e235eccc233f2d8b906ae8e4c3d717c44fcce/ci/run.sh#L19-L21), or opt-out of a default feature that enables it.
ijackson commented 7 months ago

(Comment that was here was misplaced, I entered it into the wrong MR. See https://github.com/mehcode/config-rs/pull/492#issuecomment-1795847292. Sorry.)

polarathene commented 7 months ago

Full reproduction environment to verify generated lockfile: https://github.com/mehcode/config-rs/pull/492#issuecomment-1797948314

Rebased onto #496 as requested. CI passes 👍


I've additionally:

All tests pass again 👍

Awaiting approval from either @ijackson or @matthiasbeyer

ijackson commented 7 months ago

Thanks, excellent work. It's great that we now have a non-awful method for generating a working lockfile. I have reviewed the changes and verified the lockfile creation.

The MSRV reduction is also nice.

Please do merge this.

matthiasbeyer commented 7 months ago

I am on vacation until late tomorrow. I hope I can get to having a look on my rather long backlog for this repository after tomorrow! Please do bug me if I don't!