Issue with threshold percentage

[bajjurisanthosh]

Every time my build gets success status when I run "gradle clean build", but when I generate report using "gradle cloverGenerateReport"/"gradle cloverAggregateReport" I see weird errors sometime but not always. Find below the error [ant:clover-check] The following coverage targets for null were not met: Total coverage of 77.6% did not meet target of 80% :project1:cloverGenerateReport FAILED

Note: sometime I do the same process mentioned above, but report gets generated. I don't know how to resolve. I think this type of solution isn't stable.

Find below the code I am using:

configure(allprojects - javaTestOnlyProjects()) { apply plugin: 'com.bmuschko.clover'

clover {
    initString = ".cloverbuild/clover.db"
    snapshotFile = "build/.cloverbuild/coverage.db.snapshot"
        targetPercentage = '80%'
    optimizeTests = false
    contexts {
        statement {
            name = 'log'
            regexp = '^.*LOG\\..*'
        }
        statement {
            name = 'iflog'
            regexp = '^if \\(LOG\\.is.*'
        }
        method {
            name = 'main'
            regexp = 'public static void main\\(String args\\[\\]\\).*'
        }
    }

    compiler {
        encoding = 'UTF-8' 
    }

    report {
        html = true
        //xml = true
        filter = 'log,iflog,main'
    }
}

dependencies {
    clover 'org.openclover:clover:4.2.0'                    
}       

}

Can someone help me to understand If I am missing something

[Alex-Vol]

When you use the targetPercentage setting you are asking for this behavior of the plugin. You want the build to fail hard when the coverage is below a minimum acceptable percentage. Why is that confusing to you? It works as documented in my opinion.

What is the expectation you have when the targetPercentage is not achieved by your code testing?

[bajjurisanthosh]

I would like to bypass the percentage in the build even if we have targetPercentage, can you let me know how do we achieve this. May intention is to generate report even if threshold percentage is less than TargetPercentage. Let me know if we can use a command to bypass threshold and generate report

[Alex-Vol]

Make targetPercentage very low. 0% is the lowest you can have. Then the job will generate the report and not automatically fail.

[CullenClark]

Hey Alex,

The issue in question is that the behavior is not consistent. We are fine with the target of 80%, if we consistently got 80%. It appears that each run, depending on the order of execution, the threshold is applied before the run completes and fails inappropriately. When it fully executes, the coverage is over 80%, and this reports and succeeds. Changing nothing, we can run the command again, and it will fail by a different code percentage under 80%.

If we run the command with a 0% threshold, we consistently get the same coverage. Only when applying the threshold do we get inconsistent results, seemingly because it is failing some project early depending on the order of execution.

Alternatively to this fix, is there a way to bypass the threshold via commandline? We want the developers to have 80%, which means the value has to be in the file, but we actually threshold separately as part of our CI process anyway, so this is somewhat redundant for us. We're fine with it being there if we can get consistent executions, but this would be a viable alternative for us.

Appreciate your help and thoughts.

[Alex-Vol]

It is easy to create a property with a default value in gradle.properties and override that in the command line execution. Example:

build.gradle

clover {
   targetPercentage = "${cloverTargetCoverage}%"
}

gradle.properties

cloverTargetCoverage=0

Use -PcloverTargetCoverage=80 when needed.

I wonder if the unpredictable behavior is caused by concurrency in tests. This is also possibly an issue in Clover core which means I can do nothing to correct it. All we do in this plugin is configure the Clover Ant tasks and invoke them so this does appear to be a bug in Clover. You could check for this issue in the OpenClover web site. Also, try using --max-workers 1 and remove any concurrency in unit test execution such as forking.