astj / flake8-formatter-junit-xml

JUnit XML formatter for flake8
MIT License
17 stars 10 forks source link

Pass line and file as text rather than xml attribute #11

Open dmgening opened 5 years ago

dmgening commented 5 years ago

Hi! I have trouble displaying resulting XML with Gitlab test reports feature. Currently, it only shows the source that generated the error and no information for exact place where the error has occurred.

As I understand Gitlab closely follows JUnit XML specification by IBM (or at least they bother to put a link to it from their documentation) and it does not specify file and line attributes for TestCase element. https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html

Could we have an option to write them as plain text output? Either to failure.message or failure.text would be great.

astj commented 5 years ago

Thank you for reporting. I didn't know IBM's specification!

This formatter's failure.message already includes filePath and line. ref: https://github.com/astj/flake8-formatter-junit-xml/blob/master/flake8_formatter_junit_xml/formatter.py#L28 example: https://github.com/astj/flake8-formatter-junit-xml/blob/master/tests/expected.xml#L5 (which means in file some/filename.py at line 2 and column 1).

Isn't it enough?

dmgening commented 5 years ago

Weird, it might be multiple thing overlapping then. I'll try to gather some details how xml is processed.

To be more specific here is what i see in report attached to merge request:

screenshot 2019-01-09 at 20 53 42 screenshot 2019-01-09 at 20 54 02

And here is actual XML output:

    <testsuite disabled="0" errors="0" failures="1" name="flake8._/errors_py" skipped="0" tests="1" time="0">
        <testcase file="./errors.py" line="138" name="F811, redefinition of unused 'CardLostError' from line 120">
            <failure message="./errors.py:138:1: F811 redefinition of unused 'CardLostError' from line 120" type="failure">class CardLostError(ProcessingError):
^</failure>
        </testcase>
    </testsuite>
dmgening commented 5 years ago

So I've taken some time and found actual parser Gitlab CE uses (https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/parsers/test/junit.rb), model (https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/reports/test_case.rb) and list of exposed fields (https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/serializers/test_case_entity.rb)

At this point, I think I've misled you about JUnit spec since they are gathering data really close to what you are generating :) Sorry for that. Althrough seems like JUnit team publishing XSD: https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/resources/jenkins-junit.xsd

As for my issue, I think it might be related to missing classname property. They have an opened issue for that, but I didn't had time to check if it would change the output. https://gitlab.com/gitlab-org/gitlab-ce/issues/50964