search

ZiyueWang25 / intercode

[NeurIPS 2023 D&B] Code repository for InterCode benchmark https://arxiv.org/abs/2306.14898
https://intercode-benchmark.github.io/
MIT License
0 stars 0 forks source link

[Install] pytest cannot work by directly calling it at root. #3

Open ZiyueWang25 opened 6 months ago

ZiyueWang25 commented 6 months ago

Describe the bug We need to call pytest to learn whether the agent has fixed the issue or not. However, some repository cannot work directly when calling pytest. For example, when I call pytest in the root of sqlfluff, it shows 

ImportError while loading conftest '/Users/zyw/sqlfluff/test/conftest.py'.
test/conftest.py:11: in <module>
    from sqlfluff.cli.commands import quoted_presenter
E   ModuleNotFoundError: No module named 'sqlfluff'

So it relates with the repository directory structure.

EliorFureraj commented 6 months ago

Putting what I mentioned during the meeting here for reference: Old versions of repos have a ton of issues which we can't fix manually. Best workaround is doing what SWE-bench does and run pytest only on the associated test files. We should pass the pytest command we know works to the prompt for the agent to use as well.