search

pestphp / pest

Pest is an elegant PHP testing Framework with a focus on simplicity, meticulously designed to bring back the joy of testing in PHP.
https://pestphp.com
MIT License
9.62k stars 358 forks source link

Some tests do not traverse the whole dataset #98

Closed bpolaszek closed 4 years ago

bpolaszek commented 4 years ago

Hey there,

First, thank you for the awesome work on Pest! This is the 1st time I use it on a project and it's a real delight.

However, I noticed that sometimes, for an unknown reason, the whole dataset is not entirely traversed when using the with() method.

Example here: $combinations->asArray() returns a dataset of 44 combinations (verified through that first test), but only the 8th first ones are run, as if the others simply didn't exist.

Here's the full output of the test:

   FAIL  Tests\Unit\Security\TopicMatcher\Publish\TopicMatcherTest
  ✓ it contains 44 combinations
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false)
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false)
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, true)
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, true)
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false)
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false)
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true)
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, true)
  • All combinations have been tested

  ---

  • Tests\Unit\Security\TopicMatcher\Publish\TopicMatcherTest > All combinations have been tested
  Failed asserting that 8 matches expected 44.

Any ideas?

Thank you, Ben

PS: Running Pest 0.2 (used to fail also on 1.x) on PHP 7.4.

nunomaduro commented 4 years ago

This is indeed a bug: https://github.com/pestphp/pest/blob/283d8f3e03956cbee210d69edd5ad13690078723/src/Datasets.php#L95.

Some combinations in that $combinations variable have the same short description, making pest think that the test is the same.

bpolaszek commented 4 years ago

Brilliant! Tested on dev-master:

 PASS  Tests\Unit\Security\TopicMatcher\Publish\TopicMatcherTest
  ✓ it contains 44 combinations
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #1
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #1
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #2
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #2
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #3
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #3
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #4
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, true) #1
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, true) #1
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, true) #2
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, true) #2
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, true) #3
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, true) #3
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, true) #4
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #5
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #4
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #6
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #5
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #7
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #6
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), true, false) #8
  ✓ Topic can be updated only when authorized with ('/foo/bar', Lcobucci\JWT\Token Object (...), false, false) #7
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #1
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false) #1
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #2
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false) #2
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #3
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false) #3
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #4
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #1
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, true) #1
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #2
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #5
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #3
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, true) #2
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #4
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, true) #3
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #5
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #6
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, true) #6
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #7
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false) #4
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), true, false) #8
  ✓ Topic can be updated only when authorized with ('/foo/baz', Lcobucci\JWT\Token Object (...), false, false) #5
  ✓ all combinations have been tested

Thank you @fetzi @nunomaduro for your reactivity! 🙏