search

kiwi-bdd / Kiwi

Simple BDD for iOS
BSD 3-Clause "New" or "Revised" License
4.14k stars 512 forks source link

Rare should equal failure with dictionary comparison [Remastered]. #682

Open lolgear opened 8 years ago

lolgear commented 8 years ago

Well, as I told before

dictionaries equal doesn't work well.

I would like to narrow down the area for inspection, but I could only give you a clue:

git: yourkarma/jwt branch: master ( latest release, for example, 2.0.2 ) JWTSpec.m describe encoding -> context claims set -> it decode claims set and verify it correctly.

steps to reproduce:

  1. download latest release.
  2. open JWT.xcworkspace
  3. cleanup everything.
  4. run tests.
  5. do 4 - 3 - 4 - 4 - 4 - 3 - 4 or whatever order you'd like.
  6. catch results as failure of example mentioned above.
modocache commented 8 years ago

Thanks for the remastered version! 😁

So I guess the problem is still occurring? Is it possible to reduce your test case at all? That is, does a dictionary comparison fail stochastically when not using code from the JWT project?

I ask because Kiwi uses -[NSObject isEqual:] for its equality matcher -- there isn't any special code path when comparing NSDictionary. For two dictionaries to only sometimes not be equal, I'd be forced to suspect either:

  1. All Kiwi equality comparisons fail stochastically, or...
  2. ...Apple's -[NSDictionary isEqual:] sometimes returns different values.

I think the more likely explanation is that something in the JWTSpec_encoding_claims_set_decode_claims_set_and_verify_it_correctly test has a race condition, which causes the two dictionaries to not be equal sometimes.

Of course, it's possible there's some bug I'm overlooking in Kiwi. Reducing the test case would help find that, too!

lolgear commented 8 years ago

@modocache hey, how could I isolate this test? Is it possible to move it to single spec file? I think no, because kiwi doesn't allow to run single test :( Single test project as an option?

modocache commented 8 years ago

Hey @lolgear! Yeah, I think a new Xcode project with just this test, and just the source code it's testing, will probably help isolate the problem better.

lolgear commented 8 years ago

@modocache check master branch: https://github.com/lolgear/JWT Inspection -> SherlockHolmes project. ( In spec you could find steps to reproduce issue )

lolgear commented 8 years ago

@modocache any update?

modocache commented 8 years ago

Not yet, sorry! Will try to find a spare minute during a weekend soon.

modocache commented 8 years ago

Or if you have some spare time before I do, you could take a look at the Kiwi internals to try and figure it out!