search

gitbugactions / gitbug-java

A Reproducible Benchmark of Recent Java Bugs
MIT License
27 stars 9 forks source link

Compile gitbug-java outside the environment #33

Open Coppelian opened 1 week ago

Coppelian commented 1 week ago

Hi Gitbug-java researcher,

Thank you for your excellent dataset. I have a question regarding gitbug-java and might need some insight.

I'm creating JSON files to extract bug metadata info from gitbug-java, which needs me to compile test gitbug-java bugs outside the poetry shell and docker. I want to know if I want to download the dataset and use ant or maven to compile the project, can you provide me steps to do it without using gitbug-java API?

Thank you, Coppelia

andre15silva commented 1 week ago

Hi @Coppelian ,

I want to know if I want to download the dataset and use ant or maven to compile the project, can you provide me steps to do it without using gitbug-java API?

This is not supported.

The commands to compile and execute the tests are provided in the GitHub workflows of each project, and the workflows are executed via act. This is because: 1) we want to use the environment defined by the developers of each project, 2) we must execute the bugs in this environment to ensure reproducibility.

I suggest you look into how we identify and instrument test execution commands to obtain test results. It is possible that the metadata you wish to extract can be extracted by simply further instrumenting the workflows. For example, see https://github.com/gitbugactions/gitbugactions/blob/fe5a80a0d941139a3eda82e1fb35ebe093daa50b/gitbugactions/actions/python/pytest_workflow.py#L26, for example. Our paper GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions also contains more details.

If you really must execute outside of the act environment, you might be able to extract the commands from the workflows. Be aware, however, that this will likely break reproducibility of some bugs since some third-party dependencies might no longer be available.

Best, André

Coppelian commented 1 week ago

Hi @andre15silva ,

Thank you for responding to my question!

I have some additional questions regarding gitbug-java.

This dataset contains patches for source code, test suite, and non-code part. In its API command, we can use gitbug-java to check BID WORK_DIR [--fixed]. My questions are: Can Gitbug-java patch on source code only without the test suite and non-code part? Which test suite should we use to verify our plausible patch? Since the buggy patch and the fixed patch for the test suite are different?

Thank you!

Sincerely, Chris

andre15silva commented 1 week ago

Can Gitbug-java patch on source code only without the test suite and non-code part?

When you use the checkout command the test and non-code patches are automatically applied, in both buggy and fixed scenarios.

Which test suite should we use to verify our plausible patch? Since the buggy patch and the fixed patch for the test suite are different?

You should always use the test suite after applying the test patch (if it exists). This is because the test patch includes changes to the tests that contain the desired specification.