princeton-nlp / SWE-bench

[ICLR 2024] SWE-Bench: Can Language Models Resolve Real-world Github Issues?
https://www.swebench.com
MIT License
1.5k stars 247 forks source link

matplotlib__matplotlib-18869 can't pass (?) due to test_tmpconfigdir_warning #172

Closed waterson closed 2 weeks ago

waterson commented 2 weeks ago

Describe the bug

When I look at the output logs for matplotlib__matplotib-18869, I see the following:

+ pytest --no-header -rA --tb=no -p no:cacheprovider lib/matplotlib/tests/test_matplotlib.py
============================= test session starts ==============================
collected 8 items

lib/matplotlib/tests/test_matplotlib.py ....s...                         [100%]

=========================== short test summary info ============================
PASSED lib/matplotlib/tests/test_matplotlib.py::test_parse_to_version_info[3.5.0-version_tuple0]
PASSED lib/matplotlib/tests/test_matplotlib.py::test_parse_to_version_info[3.5.0rc2-version_tuple1]
PASSED lib/matplotlib/tests/test_matplotlib.py::test_parse_to_version_info[3.5.0.dev820+g6768ef8c4c-version_tuple2]
PASSED lib/matplotlib/tests/test_matplotlib.py::test_parse_to_version_info[3.5.0.post820+g6768ef8c4c-version_tuple3]
PASSED lib/matplotlib/tests/test_matplotlib.py::test_importable_with_no_home
PASSED lib/matplotlib/tests/test_matplotlib.py::test_use_doc_standard_backends
PASSED lib/matplotlib/tests/test_matplotlib.py::test_importable_with__OO
SKIPPED [1] lib/matplotlib/tests/test_matplotlib.py:20: chmod() doesn't work as root
========================= 7 passed, 1 skipped in 9.47s =========================

But this instance fails to pass because of test_tmpconfigdir_warning.

            "PASS_TO_PASS": {
                "success": [
                    "lib/matplotlib/tests/test_matplotlib.py::test_importable_with_no_home",
                    "lib/matplotlib/tests/test_matplotlib.py::test_use_doc_standard_backends",
                    "lib/matplotlib/tests/test_matplotlib.py::test_importable_with__OO"
                ],
                "failure": [
                    "lib/matplotlib/tests/test_matplotlib.py::test_tmpconfigdir_warning"
                ]
            },

Since the test is never run, should this be removed from the PASS_TO_PASS for this instance?

Steps/Code to Reproduce

Run evaluation for `matplotlib__matplotlib-18869 using SWE-bench_Lite

Expected Results

Instance passes given the above output

Actual Results

Instance fails because test_tmpconfigdir_warning is never run

System Information

No response

waterson commented 2 weeks ago

Oops... running with an older version here. Sorry!