test.test_result_test TestResultStateTest is unstable

[mrtyler]

Latest master with Python 2.7.5. When I do make test I get this failure:

*snip*
test.test_program_test ParseTestRunnerCommandLineArgsTest.test_parse_test_runner_command_line_args_rerun_test_file ... ok in 0.00s
test.test_result_serializable_test TestResultIsSerializableTestCase.test_not_garbled_by_serialization ... ok in 0.00s
test.test_result_serializable_test TestResultIsSerializableTestCase.test_test_result_is_serializable ... ok in 0.00s
test.test_result_test TestResultStateTest.test_results ... fail: test.test_result_test TestResultStateTest.test_results
Traceback (most recent call last):
  File "./test/test_result_test.py", line 112, in test_results
    assert_equal([result.success for result in test_results], [False, True])
AssertionError: assertion failed: l == r
l: [True, False]
r: [False, True]

Diff:
l: [True, False]
r: [False, True]

FAIL in 0.00s
*snip*

But when I run the test more surgically, it always passes:

[tyler@toaster:~/Testify]$ PYTHONPATH=~/Testify ~/Testify/bin/testify -v --summary --exclude-suite=catbox test.test_result_test TestResultStateTest.test_results
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py:979: UserWarning: /Users/tyler/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
test.test_result_test TestResultStateTest.test_results ... ok in 0.00s

========================================================================
FAILURES
None!

PASSED.  1 test / 1 case: 1 passed, 0 failed.  (Total test time 0.00s)

[tyler@toaster:~/Testify]$ PYTHONPATH=~/Testify ~/Testify/bin/testify -v --summary --exclude-suite=catbox test.test_result_test TestResultStateTest
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py:979: UserWarning: /Users/tyler/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
test.test_result_test TestResultStateTest.test_results ... ok in 0.00s

========================================================================
FAILURES
None!

PASSED.  1 test / 1 case: 1 passed, 0 failed.  (Total test time 0.00s)
[tyler@toaster:~/Testify]$ PYTHONPATH=~/Testify ~/Testify/bin/testify -v --summary --exclude-suite=catbox test.test_result_test
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources.py:979: UserWarning: /Users/tyler/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
  warnings.warn(msg, UserWarning)
test.test_result_test TestResultStateTest.test_results ... ok in 0.00s
test.test_result_test TestResultTestCase.test_format_exception_info_assertion ... ok in 0.00s
test.test_result_test TestResultTestCase.test_format_exception_info_assertion_pretty ... ok in 0.00s
test.test_result_test TestResultTestCase.test_format_exception_info_error ... ok in 0.00s
test.test_result_test TestResultTestCase.test_format_exception_info_error_pretty ... ok in 0.00s
test.test_result_test TestResultTestCase.test_format_exception_info_multiple ... ok in 0.00s

========================================================================
FAILURES
None!

PASSED.  6 tests / 2 cases: 6 passed, 0 failed.  (Total test time 0.01s)

$ git bisect start
$  git bisect good c115aa64b64220770aca14be0735462d1fdf936d
$  git bisect bad origin/master
$  PYTHONPATH=~/Testify git bisect run ~/Testify/bin/testify -v --summary --exclude-suite=catbox test

*snip*

dc45d63482ad7d7a1659878bfa5af9d5e078d7f9 is the first bad commit
commit dc45d63482ad7d7a1659878bfa5af9d5e078d7f9
Author: @blampe
Date:   Wed Jun 12 11:42:48 2013 -0700

    make test case instances report correct test results

:040000 040000 66cfa86f90416205fd7b91239fa11f16207eb0cb 08b9596f3f0bbc7cd1303959596c708f1eaba024 M  test
:040000 040000 4c848212dc2442094f15f164bcbd0010739958b9 f5dfa4f5fe1981dbedcf922d6ef2815e1c796700 M  testify
bisect run success

[blampe]

My guess is something going on with the ordering of results returned. IIRC that test_results method is just returning the values of a dict, which aren't guaranteed to be in any particular order. I'll confirm tomorrow and put together a fix.