search

rjust / defects4j

A Database of Real Faults and an Experimental Infrastructure to Enable Controlled Experiments in Software Engineering Research
MIT License
705 stars 300 forks source link

Extract build configurations for `defects4j test` for each project #593

Open StwayneXG opened 2 weeks ago

StwayneXG commented 2 weeks ago

I was adding a self-made test case to see if they work fine with the defects4j test API. However, I run into errors that this feature is not available with source 1.4. (for example try-catch with resources).

Also I wanted to extract the version of junit to understand if @Test could be added to the file or not.

I was looking into build files mentioned in the query but they don't have this information and that lead me to build.xml files specific to each folder. But I still didn't figure it out. Let me know which files to parse to get this information.

rjust commented 1 week ago

Hi @StwayneXG,

Can you please detail the steps that you took to create your own test and to compile and run it?

Generally speaking, you should run your own tests with defects4j test -s. These "external" tests are compiled without setting a particular source or target version.

If you write "external" tests, you can write JUnit4 tests and you do not have to worry about how the project version is built. For Defects4J version 2, you can use all language features available in Java 8; for version 3, you can use all language features available in Java 11.

Hope this helps.

Best, René

StwayneXG commented 1 week ago

Hi,

So, I basically added a testcase inside one of the existing test case files (e.g. there is a file called TestCommons.java). I run it with defects4j test -t org.java.commons.TestCommons:Testfunction.

Also, when you say version 3 of defects4j, I don't understand which one you are referring to as from the tags of the repository, I only see versions up till 2.0.1.

Thanks for suggesting defects4j test -s. I'll look more into it. Do you have any example of how to use it for any project? That would be really helpful.

Thanks, Irtaza

rjust commented 1 week ago

Here is a concrete example for Lang-1:

defects4j checkout -p Lang -v 1f -w /tmp/Lang-1f
cd /tmp/Lang-1f
tar -C src/test/java -cjf Lang-1f-my_manual_tests.1.tar.bz2 org/apache/commons/lang3/SystemUtilsTest.java
tar -tf Lang-1f-my_manual_tests.1.tar.bz2
defects4j test -s Lang-1f-my_manual_tests.1.tar.bz2

What do these commands do?

  1. Checkout Lang-1
  2. Change to working directory
  3. Create a test suite archive (see naming convention). In this example, the archive simply includes a test class from the developer-written tests, but you can put arbitrary .java files in the archive.
  4. List the content of the test suite archive
  5. Compile and run all tests in the test suite archive

The test command extracts the archive into the working directory, compiles the sources with the project classes and dependencies on the classpath, and then calls JUnit to execute the provided test cases.

Version 3 is in java-11-compatibility. We will merge this branch in the next few days and officially release version 3.