jacoco / jacoco

:microscope: Java Code Coverage Library
https://www.jacoco.org/jacoco/
Other
4.17k stars 1.14k forks source link

JaCoCo filter for Generated uses wildcard, filters any annotation that has the word "Generated" in it #976

Open jdsmithsos opened 4 years ago

jdsmithsos commented 4 years ago

THIS IS A BUG TRACKER ONLY. FOR QUESTIONS PLEASE CHECK FAQ OR USE FORUM:

http://www.jacoco.org/jacoco/trunk/doc/faq.html

https://groups.google.com/forum/?fromgroups=#!forum/jacoco

Please understand that ISSUES WITHOUT FOLLOWING INFORMATION WILL BE CLOSED WITHOUT COMMENTS! Thank you for filling bug report!

Steps to reproduce

  1. With a Java project, identify a class that will be used to reproduce this bug
  2. Write a JUnit test class to get 100% code coverage. This can easily be done with a single method that does a System.out.println("print something");
  3. Verify that the JUnit test gets 100% code coverage on the class being tested, using Eclipse EclEmma plugin (Coverage As)
  4. Create a class annotation named something like TestGeneratedTheory
  5. Use the TestGeneratedTheory annotation to annotate the class being tested
  6. Run code coverage again, and note that the class being tested is not listed in the code coverage report

Expected behaviour

Just because an annotation has "Generated" in the name doesn't mean that it should be filtered from code coverage. The documentation that I could find for this filter feature of JaCoCo specified that the annotation name itself should only be named "Generated", which makes sense.

To make the filter even more explicit, and meaningful, maybe an annotation name like "JaCoCoIgnore" should be used. Also, JaCoCo could provide the filter annotations in a separate library.

Actual behaviour

Any annotation that has "Generated" in the name is filtered and not included in the code coverage.

(In case of exceptions provide FULL STACKTRACE)

// class without TestGeneratedTheory annotation public class TestThisCodeCoverage {

public void testThisMethod() { System.out.println("in testThisMethod"); }

}

// test class to test TestThisCodeCoverage public class TestThisCodeCoverageTest {

@Test public void testThisMethodTest() { new TestThisCodeCoverage().testThisMethod(); }

}

// annotation class to test the "Generated" theory of filtering when an annotation has "Generated" in the name @Retention(RUNTIME) @Target(TYPE) public @interface TestGeneratedTheory {

}

// class with TestGeneratedTheory annotation // because this annotation has "Generated" in the name, JaCoCo will filter this class in code coverage @TestGeneratedTheory public class TestThisCodeCoverage {

public void testThisMethod() { System.out.println("in testThisMethod"); }

}

richardforrestbarker commented 2 years ago

I think this is the best place for my report on a similar Issue.

Steps to reproduce JaCoCo version: 0.8.7 Operating system: Mac OS Tool integration: Gradle, IntelliJ Complete executable reproducer:

Steps: Using a project correctly configured with JaCoCo, with several packages and classes. Clean and Build the project Check that all expected packages are included in JaCoCo's report. On a single class, create / use an annontation meeting the requirements to trigger JaCoCo to ignore it for code coverage, in any package exoected to exist in the report. Clean and Build the project. Check that the expected packages are included in JaCoCo's report and note that the package containing the annontated class and all of its classes are missing.

Expected behaviour Given an annotation with the word Generated in it's name on a given class in a given package, when JaCoCo generates reports, then only the class annontated as such must be ignored for the purposes of the report, and nothing else.

Actual behaviour Entire package is filtered from code coverage, not just the class.

Basically class A and class B are in package X; class C is in package X.Y; and class D is in package Z. When class A is annontated with a runtime or class retained annotation containing Generated in it's name (like lombok's lombok.Generated which is retained at runtime) then all classes in package X are ignored. Packages X.Y and Z are not ignored in this case which is expected.

Godin commented 2 years ago

I think this is the best place for my report on a similar Issue.

@richardforrestbarker First of all - no, because you're describing different issue. Moreover following your steps I and many other people receive expected behavior. So please create new ticket and instead of describing steps please attach complete executable reproducer.

0d00 commented 2 years ago

I would like to work on this issue. May i have a try?

marchof commented 2 years ago

Not quite whether the proposed solution will work for our existing use cases. If we create our own annotation this means the code base needs a dependency on that library containing the annotation. Not sure whether this is desired.

Also we would convince frameworks like Lombok to use this. This is very unlikely.

0d00 commented 2 years ago

@marchof Thank you for your reply! I understand your meaning after thinking twice of the issue. It is really a tough one. Maybe i'd better find an easier one. By the way, thank again for your response, i never thought i could receive a response so soon.

XenoAmess commented 2 years ago

Hi. Is it better for jacoco open a configuration like "generatedAnnotationList", and its default value be lombok.Generated, and we can add more annotations by ourselfs in project configuration... I see the codes of LombokFillter, and I think it techniqually doable. I can do it if you want.

marchof commented 2 years ago

@XenoAmess It is not a question whether it can be technically implemented. It is a question how to maintain it. Filtering is quite a complex topic and config options add more dimensions to it (beside Java Compiler Versions etc). Therefore we decided to not offer any config options for filtering.

knutwannheden commented 1 year ago

In relation to this issue: The latest IntelliJ IDEA EAP version allows specifying annotations which will exclude methods from their coverage report: https://blog.jetbrains.com/idea/2022/10/intellij-idea-2022-3-eap-2/

marchof commented 1 year ago

@knutwannheden Thanks for the link! As you can see in their screenshots there is a new configuration option to configure the names of the annotations. Something we want to avoid, see above.

knutwannheden commented 1 year ago

Yes, I gathered as much. But I think it is good to know, because if JaCoCo decides on a single annotation, then users who want to use IDEA's code coverage while working in the IDE, can configure it to use JaCoCo's annotation.