rspec / rspec-support

Common code needed by the other RSpec gems. Not intended for direct use.
https://rspec.info
MIT License
99 stars 103 forks source link

Diff reports confusing output when used with "fuzzy" matchers like `anything` #551

Open Epigene opened 1 year ago

Epigene commented 1 year ago

Subject of the issue

Using .to match with a complex expectation that mixes pure values with "fuzzy" matchers like anything produces false positives in Diff, obfuscating which key actually differs from the expectation.
I don't know whether this is a problem with match or anything/*_including.

Your environment

Steps to reproduce

# frozen_string_literal: true

begin
  require "bundler/inline"
rescue LoadError => e
  $stderr.puts "Bundler version 1.10 or later is required. Please update your Bundler"
  raise e
end

gemfile(true) do
  source "https://rubygems.org"

  gem "rspec", "3.11.0"
  gem "rspec-expectations", "3.11.0"
  gem "rspec-mocks", "3.11.1"
end

puts "Ruby version is: #{RUBY_VERSION}"
require 'rspec/autorun'

RSpec.describe 'false positive in #match diff' do
  it 'reports excess rows as mismatching' do
    expect(
      a: :a,
      b: [:b, :bb],
      c: :c,
      d: :d,
    ).to match(
      a: anything,
      b: array_including(:b),
      c: :c,     
      d: :actual_mismatch,
    )
  end
end

Output:

$ ruby rspec_demo.rb
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Using bundler 2.3.20
Using diff-lcs 1.5.0
Using rspec-support 3.11.1
Using rspec-core 3.11.0
Using rspec-expectations 3.11.0
Using rspec-mocks 3.11.1
Using rspec 3.11.0
Ruby version is: 2.7.5
F

Expected behavior

I'm expecting that Diff part will highlight only the key(s) actually failing their asserts, which is only :d.

Failures:

  1) false positive in #match diff reports excess rows as mismatching
     Failure/Error:
       expect(
         a: :a,
         b: [:b, :bb],
         c: :c,
         d: :d,
       ).to match(
         a: anything,
         b: array_including(:b),
         c: :c,
         d: :actual_mismatch,

       expected {:a=>:a, :b=>[:b, :bb], :c=>:c, :d=>:d} to match {:a=>#<RSpec::Mocks::ArgumentMatchers::AnyArgMatcher:0x00007ff34d667f18>, :b=>#<RSpec::Mocks::ArgumentMatchers::ArrayIncludingMatcher:0x00007ff34d58cc88 @expected=[:b]>, :c=>:c, :d=>:actual_mismatch}
       Diff:

       @@ -1,5 +1,5 @@
       -:d => :actual_mismatch,
       +:d => :d,

     # rspec_demo.rb:23:in `block (2 levels) in <main>'

Actual behavior

Unfortunately, keys :a and :b are also highlighted, likely due to them using "fuzzy" matchers.

Failures:

  1) false positive in #match diff reports excess rows as mismatching
     Failure/Error:
       expect(
         a: :a,
         b: [:b, :bb],
         c: :c,
         d: :d,
       ).to match(
         a: anything,
         b: array_including(:b),
         c: :c,
         d: :actual_mismatch,

       expected {:a=>:a, :b=>[:b, :bb], :c=>:c, :d=>:d} to match {:a=>#<RSpec::Mocks::ArgumentMatchers::AnyArgMatcher:0x00007ff34d667f18>, :b=>#<RSpec::Mocks::ArgumentMatchers::ArrayIncludingMatcher:0x00007ff34d58cc88 @expected=[:b]>, :c=>:c, :d=>:actual_mismatch}
       Diff:

       @@ -1,5 +1,5 @@
       -:a => anything,
       -:b => array_including(b),
       +:a => :a,
       +:b => [:b, :bb],
        :c => :c,
       -:d => :actual_mismatch,
       +:d => :d,

     # rspec_demo.rb:23:in `block (2 levels) in <main>'
JonRowe commented 1 year ago

:wave: Hi, this is a known limitation of our differ which essentially only uses strings to produce diffs, so its unaware that the two sides did match it just sees the difference in the output used, theres not much we can do about it without a complete rewrite of the differ which is something we are interested in but has yet to find someone with enough time to finish it, I've transferred the issue to rspec-support which provides this functionality and renamed it slightly to better keep track of it, it related to #365 and #518.