Semver errors when exposing types from 3rd party crates that are not at the latest version

Scenario:

Crate A has a dependency on crate B = "1.0" and exposes crate B's types in its own public API.
The lockfile for crate A currently using B version 1.0. Version 1.1 of B exists and contains additional pub methods on its pub types.
The current version of A is semver-checked using the default (registry-based) baseline:
- The current rustdoc will use B v1.0 and will show the v1.0 methods from B as part of A's public API.
- The baseline rustdoc for A will be generated, which will cause a new lockfile to be created with the latest allowed version of B: v1.1.
- The baseline's public API for A will show additional methods that don't exist in the current version: those added in B v1.1 but not present in `B v1.0.
- cargo-semver-checks will report those as major breaking changes.

Steps to reproduce the bug with the above code

Checkout https://github.com/servo/core-foundation-rs/tree/cocoa-v0.24.1/cocoa Run rustup run nightly cargo semver-checks check-release

Actual Behaviour

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-check/tree/v0.12.0/src/queries/inherent_method_missing.ron

Failed in:
  NSEventMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSWindowOcclusionState::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSEventSwipeTrackingOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSApplicationPresentationOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSWindowOrderingMode::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  AutoresizingMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  CornerMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  NSWindowCollectionBehavior::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSEventPhase::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSAlignmentOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSTouchPhase::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSWindowStyleMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSEventModifierFlags::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  EdgeAntialiasingMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
       Final [   2.753s] semver requires new major version: 1 major and 0 minor checks failed

Expected Behaviour

No changes should be found when no changes have been made

Additional Context

It seems like things are getting confused by the bitflags macro

Debug Output

No response

Thanks for the bug report, and apologies for the inconvenience.

It's either the bitflags macro, or the OS-specific cfg attribute that only includes appkit.rs and quartzcore.rs on macOS (the two files where all the errors come from): https://github.com/servo/core-foundation-rs/blob/cocoa-v0.24.1/cocoa/src/lib.rs#L26-L31

The latter hypothesis would make this similar to #147. If so, the same workaround might apply: instead of using the index to find the previous version (the default behavior), explicitly a give a git revision to check against: https://github.com/obi1kenobi/cargo-semver-check#does-the-crate-im-checking-have-to-be-published-on-cratesio

I'd love to know if the workaround did the trick! In the meantime, I'll look at the rustdoc JSON output.

I'm having a bit of trouble reproducing this bug, on both v0.12.0 (the version you seemed to be using) and v0.12.1 (the most recent release), on both stable Rust and nightly:

$ cargo semver-checks check-release --package cocoa
    Updating index
     Parsing cocoa v0.24.1 (current)
     Parsing cocoa v0.24.1 (baseline)
    Checking cocoa v0.24.1 -> v0.24.1 (no change)
   Completed [   2.639s] 20 checks; 20 passed, 0 skipped

Here are a few more pieces of information that you might be able to provide that would be a big help in tracking this issue down:

the output of cargo semver-checks --bugreport
the full output of cargo semver-checks check-release --verbose, not just the portion that describes the semver issues discovered
two rustdoc JSON files in your target directory (the one where build output goes) — they should be at paths like target/semver-checks/target/doc/cocoa.json and `target/semver-checks/registry-cocoa-0_24_1/target/semver-checks/target/doc/cocoa.json

Software version

cargo-semver-checks 0.12.1

Operating system

Mac OS X 10.15.6 (Darwin 19.6.0)

Command-line

/Users/jrmuizel/.cargo/bin/cargo-semver-checks semver-checks --bugreport

cargo nightly version

> cargo +nightly -V
cargo 1.64.0-nightly (85b500cca 2022-07-24)

Compile time information

Profile: release
Target triple: x86_64-apple-darwin
Family: unix
OS: macos
Architecture: x86_64
Pointer width: 64
Endian: little
CPU features: fxsr,sse,sse2,sse3,ssse3
Host: x86_64-apple-darwin

$ rustup run nightly cargo semver-checks check-release -v
    Updating index
     Parsing cocoa v0.24.1 (current)
warning: unused boxed boxed `Fn` trait object that must be used
   --> core-graphics/src/event.rs:508:17
    |
508 |                 Box::from_raw(cbr);
    |                 ^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: closures are lazy and do nothing unless called

warning: anonymous parameters are deprecated and will be removed in the next edition
   --> cocoa-foundation/src/foundation.rs:614:37
    |
614 |     unsafe fn isEqualToString(self, &str) -> bool;
    |                                     ^^^^ help: try naming the parameter or explicitly ignoring it: `_: &str`
    |
    = note: `#[warn(anonymous_parameters)]` on by default
    = warning: this is accepted in the current edition (Rust 2015) but is a hard error in Rust 2018!
    = note: for more information, see issue #41686 <https://github.com/rust-lang/rust/issues/41686>

warning: `core-graphics` (lib) generated 1 warning
warning: `cocoa-foundation` (lib) generated 1 warning
 Documenting cocoa v0.24.1 (/Users/jrmuizel/src/core-foundation-rs/cocoa)
    Finished dev [unoptimized + debuginfo] target(s) in 1.20s
     Parsing cocoa v0.24.1 (baseline)
 Documenting cocoa v0.24.1
    Finished dev [unoptimized + debuginfo] target(s) in 1.20s
    Checking cocoa v0.24.1 -> v0.24.1 (no change)
    Starting 20 checks, 0 skipped
        PASS [   0.389s]       major        auto_trait_impl_removed
        PASS [   0.322s]       major        derive_trait_impl_removed
        PASS [   0.001s]       major        enum_marked_non_exhaustive
        PASS [   0.058s]       major        enum_missing
        PASS [   0.001s]       major        enum_repr_c_removed
        PASS [   0.049s]       major        enum_repr_int_changed
        PASS [   0.049s]       major        enum_repr_int_removed
        PASS [   0.002s]       major        enum_struct_variant_field_missing
        PASS [   0.373s]       major        enum_variant_added
        PASS [   0.361s]       major        enum_variant_missing
        PASS [   0.036s]       major        function_missing
        FAIL [   0.008s]       major        inherent_method_missing
        PASS [   0.071s]       major        sized_impl_removed
        PASS [   0.001s]       major        struct_marked_non_exhaustive
        PASS [   0.032s]       major        struct_missing
        PASS [   0.042s]       major        struct_pub_field_missing
        PASS [   0.006s]       major        struct_repr_c_removed
        PASS [   0.002s]       major        struct_repr_transparent_removed
        PASS [   0.001s]       major        unit_struct_changed_kind
        PASS [   0.368s]       major        variant_marked_non_exhaustive
   Completed [   2.173s] 20 checks; 19 passed, 1 failed, 0 skipped

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-check/tree/v0.12.1/src/queries/inherent_method_missing.ron

Failed in:
  NSWindowOrderingMode::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSWindowOrderingMode::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:252
  NSAlignmentOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSAlignmentOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:260
  NSEventMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2334
  NSEventPhase::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSEventPhase::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2273
  NSApplicationPresentationOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  NSApplicationPresentationOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:185
  CornerMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  CornerMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1350
  NSWindowCollectionBehavior::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSWindowCollectionBehavior::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:878
  NSEventModifierFlags::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSEventModifierFlags::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2375
  NSTouchPhase::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSTouchPhase::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2285
  NSEventSwipeTrackingOptions::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  NSEventSwipeTrackingOptions::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:2259
  AutoresizingMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  AutoresizingMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1359
  EdgeAntialiasingMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  EdgeAntialiasingMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/quartzcore.rs:1341
  NSWindowStyleMask::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowStyleMask::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:203
  NSWindowOcclusionState::from_bits_unchecked, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::intersection, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::union, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::symmetric_difference, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
  NSWindowOcclusionState::complement, previously in file /Users/jrmuizel/.cargo/registry/src/github.com-1ecc6299db9ec823/cocoa-0.24.1/src/appkit.rs:896
       Final [   2.759s] semver requires new major version: 1 major and 0 minor checks failed

cocoa.zip

Thank you so much! And since cargo-semver-checks depends on the underlying information generated by Rust's own rustdoc tool, would you mind upgrading to 1.64 or 1.65 stable and re-running the command? Right now I see you're running it on 1.64.0-nightly (85b500cca 2022-07-24) and I'd like to rule out any bugs internal to rustdoc that might be propagating into cargo-semver-checks' own analysis and which might have already been fixed between 1.64.0-nightly (85b500cca 2022-07-24) and 1.64/1.65 stable.

I get the same on 1.64 stable.

and on rustc 1.67.0-nightly (95a3a7277 2022-10-31)

Fascinating, thanks for confirming. I'm working to get to the bottom of this and will post more info here when I have it.

Apologies once again for the inconvenience — this is not the kind of user experience we want in cargo-semver-checks, so thank you for bearing with us.

What OS did you try to reproduce it on?

macOS 10.15.7 on an x86 CPU, almost identical to your system (10.15.6 on x86)

So, if I run it on a fresh checkout I'm also not able to reproduce.

I guess it could be a difference in the Cargo.lock file.

If you run cargo update -p bitflags --precise 1.0.4 before cargo semver-checks check-release you should be able to reproduce.

Ah thank you for narrowing down the exact details. I'll try that!

Sorry for the delay. In the last couple of weeks I lost my ability to test on a macOS device, and I was having difficulty even compiling cocoa on Linux, which slowed things down quite a bit.

After a bit of investigation, it appears that the reported errors are correct i.e. true positives:

The bitflags version bound in cocoa v0.24.1 is set to 1.0 (link), which per cargo means >=1.0,<2 and by default is currently resolved as 1.3.2. Therefore, the baseline rustdoc JSON uses bitflags v1.3.2 as a dependency.
bitflags-generated structs are part of the public API interface of cocoa v0.24.1, so all of those structs' public methods are part of cocoa v0.24.1's own public API, even though they come from a 3rd party dependency.
Running cargo update -p bitflags --precise 1.0.4 before semver-checking downgrades the local bitflags version from 1.3.2 to 1.0.4. This is what rustdoc uses to generate "current" rustdoc JSON which is compared against the baseline for semver compliance.
This is the list of methods that bitflags v1.0.4 creates on the structs it generates. This is the list of methods that bitflags v1.3.2 creates on the structs it generates.
Comparing the two lists, we see that bitflags v1.3.2 creates additional methods on its structs compared to bitflags v1.0.4: from_bits_unchecked, intersection, union, difference, symmetric_difference, complement. Note that this list matches the methods cargo-semver-checks reported as missing in the "current" version relative to the baseline — they are indeed missing because of the semver-incompatible bitflags version downgrade.

To recap: downgrading bitflags to v1.0.4 is not a semver-compatible change in cocoa, since it removes methods that were previously available in the public API under the prior "1.0" requirement bound on bitflags. cargo-semver-checks's analysis correctly identifies and reports this, so based on the present information this "works as expected" and does not appear to be a bug.

Please let me know your thoughts!

Would it be possible to build the current version without using the local Cargo.lock file? It feels like that's more representative of the actual usage scenario for a crate as a dependency.

Unfortunately, I don't think so. The current version rustdoc information (which cargo-semver-checks uses as input) is built by cargo doc which relies on the local Cargo.lock file. If cargo-semver-checks ran a cargo update command to move all the dependency versions forward in that cargo doc call, that would also result in an edited Cargo.lock file which I feel users would find unexpected and undesirable.

You may be able to work around this on your end by running something like:

cargo update
cargo semver-checks check-release <any flags you might need>
git checkout main Cargo.lock

It's also worth considering that the semver issue uncovered here can also arise in actual usage as well:

Add dependency on cocoa.
Later, add dependency on another crate that upper-bounds bitflags to an older version, like 1.0.4.
Resolution still succeeds, and chooses to install bitflags v1.0.4.
The project fails to compile, because it was accessing cocoa, getting back a bitflags type, and then calling a method on it that was added later than bitflags v1.0.4.

If the above scenario is a concern, it may be worth specifying a newer minimum version bound on cocoa's bitflags dependency to avoid it.

Is there anything else I could do to help you out with this? Were you able to adopt cargo-semver-checks for your project?

If not, I'd love to work with you to figure out how we can do better in the future.

I filed #193 about another issue I'm having.

Thanks! If it's okay with you, I'd like to close this one and continue our conversation in #193.

I just had an idea that might help solve this problem rather elegantly, if it works.

We currently generate the "current" rustdoc JSON directly from the crate's codebase itself, which makes that rustdoc subject to the codebase's lockfile. The "baseline" rustdoc is generated via its own temporary crate that declares an exact dependency on the baseline version, using a new lockfile which therefore may pull in newer versions of dependencies.

Could we generate the "current" rustdoc JSON via a temporary crate that declares a path dependency on the current version but as a result also uses a freshly-generated lockfile?

To avoid an analogous problem with git and local path baselines, we should consider using a temp crate + path dependency for those baseline types as well. In general, if this approach works then we should generate all rustdoc JSONs through a temp crate.

cc @tonowak — what do you think?

Hey @jrmuizel! If you have a chance, could you check if the current version on main resolves this issue successfully?

@tonowak and I just merged what we believe should be a fix, but we don't have a way to test it since neither of us has an Apple device nor a lockfile with non-latest dependency versions. We'd love to know if it indeed fixed the issue or if we need to do something else as well to get this use case unblocked.

It does resolve the issue.

When I run main cargo-semver-checks on cocoa-v0.24.1 with bitflags flags set to v1.0.4 in my Cargo.lock it correctly returns no changes.

Awesome, thanks for confirming and for bearing with us while we worked on it! The fix will be published in v0.18 sometime in this coming week.

Are there any other blockers toward adopting cargo-semver-checks in that repo that we can help with?

Fixed by #341, released as v0.18.0.

Apparently making a separate manifest and taking a path dependency on the library to check is unreliable too. In #900, we're seeing the path dependency use an outdated version of tokio for the "current" arm, which results in a phantom SemVer violation due to upgraded guarantees provided by the newer tokio in the baseline.

We have to run cargo update after taking the path dependency in order to be truly sure this problem is solved.

The new issue is tracked by #902.

obi1kenobi / cargo-semver-checks