search

python / mypy

Optional static typing for Python
https://www.mypy-lang.org/
Other
18.43k stars 2.82k forks source link

JUnit report jenkins compatibility #10102

Open zerkms opened 3 years ago

zerkms commented 3 years ago

Bug Report

It looks like the current stable mypy-0.800-cp36-cp36m-manylinux2010_x86_64.whl does not generate a "valid" junit compatible file as jenkins treats it. I know that the junit format is not really standardised, but I think it would be cool if the file generated was jenkins compatible (or may be a little tool was provided that converts the "incorrect" format to a "correct" one).

An example of a file that fails in jenkins is

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="1" time="3.867">
  <testcase classname="mypy" file="mypy" line="1" name="mypy-py3_6-linux" time="3.867">
    <failure message="mypy produced messages">xxx.py:62: error: Incompatible types in assignment (expression has type "None", variable has type "Union[aaa, bbb]")</failure>
  </testcase>
</testsuite>

and the corresponding stacktrace:

00:01:59.084  Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.51.5.215/10.51.5.215:40124
00:01:59.084        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
00:01:59.084        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
00:01:59.084        at hudson.remoting.Channel.call(Channel.java:1001)
00:01:59.084        at hudson.FilePath.act(FilePath.java:1157)
00:01:59.084        at hudson.FilePath.act(FilePath.java:1146)
00:01:59.084        at org.jenkinsci.plugins.xunit.XUnitProcessor.processTestsReport(XUnitProcessor.java:183)
00:01:59.084        at org.jenkinsci.plugins.xunit.XUnitProcessor.process(XUnitProcessor.java:157)
00:01:59.084        at org.jenkinsci.plugins.xunit.pipeline.XUnitResultsStepExecution.run(XUnitResultsStepExecution.java:83)
00:01:59.084        at org.jenkinsci.plugins.xunit.pipeline.XUnitResultsStepExecution.run(XUnitResultsStepExecution.java:49)
00:01:59.084        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
00:01:59.084        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
00:01:59.084        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
00:01:59.084        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
00:01:59.084        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
00:01:59.084  org.jenkinsci.plugins.xunit.service.TransformerException: The result file '/home/jenkins/agent/workspace/project/.ci/logs/mypy.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
00:01:59.084    at org.jenkinsci.plugins.xunit.service.XUnitTransformerCallable.invoke(XUnitTransformerCallable.java:111)
00:01:59.084    at org.jenkinsci.plugins.xunit.service.XUnitTransformerCallable.invoke(XUnitTransformerCallable.java:38)
00:01:59.084    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3313)
00:01:59.084    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
00:01:59.084    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
00:01:59.084    at hudson.remoting.Request$2.run(Request.java:376)
00:01:59.084    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
00:01:59.084    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
00:01:59.084    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
00:01:59.084    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
00:01:59.084    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:119)
00:01:59.084    at java.lang.Thread.run(Thread.java:748)

To Reproduce

(Write your steps here:)

  1. Generate a junit report with --junit-xml
  2. Feed it to the jenkins junit

Expected Behavior

(Write what you thought would happen.)

Actual Behavior

(Write what happened.)

Your Environment

zerkms commented 3 years ago

UPD: I believe this would be the jenkins' junit xsd: https://github.com/junit-team/junit5/blob/main/platform-tests/src/test/resources/jenkins-junit.xsd

srittau commented 3 years ago

As far as I can tell, Jenkins is expecting a root <testsuites> element, which then contains individual <testsuite> elements.

zerkms commented 3 years ago

I have borrowed the core of the solution https://github.com/xmlrunner/unittest-xml-reporting/blob/master/xmlrunner/extra/xunit_plugin.py does in the similar case with a little alteration.

The transform that makes mypy report valid for me looks like this

TRANSFORM = etree.XSLT(etree.XML(b'''\
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:output method="xml" indent="yes" />

    <!-- /dev/null for these attributes -->
    <xsl:template match="//testcase/@file" />
    <xsl:template match="//testcase/@line" />
    <xsl:template match="//testcase/@timestamp" />

    <xsl:template match="@skips">
       <xsl:attribute name="skipped">
          <xsl:value-of select="."/>
       </xsl:attribute>
    </xsl:template>

    <!-- copy the rest -->
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>'''))

Which does the same as the xmlrunner one + a rename of skips attribute to skipped.

PeterJCLaw commented 1 year ago

Given the discussion on https://github.com/python/mypy/issues/4233 indicated that it would be preferable to move the XML output generation to external tools, is this issue like to be worked on? (I don't mind if it is, however if it does get fixed then I'd vote for re-opening & fixing https://github.com/python/mypy/issues/4233 too).

Alternatively, perhaps it's time to just remove the --junit-xml flag and point people to other tools for their XML needs? (I've personally found that https://pypi.org/project/mypy2junit/ works better than --junit-xml, there may be others)