Exception on malformed XML raised by oelint when a recipe contains multi lined inherit

[fmartinsons]

I have an exception when using oelint module on one of my recipe which use multilined inherit, e.g which contains something like:

inherit autotools \
        gobject-introspection \
        pkgconfig \

I used warrior branch (the sha1 of meta-sca is eb387d0ff275e1b27c3532548b75b8bd791c8306), this multiline writing trigg the following exception in meta-sca (when SCA_SCOPE_FILTER contains style):

ERROR: sfxtpm-v1.0.0+gitAUTOINC+07acc0811f-r0 do_sca_oelint_core: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_sca_oelint_core(d)
     0003:
File: '/home/fmartinsons/TAPOS_build/tapos/meta-sca/classes/sca-oelint-core.bbclass', lineno: 131, function: do_sca_oelint_core
     0127:    dm_output = do_sca_conv_oelint(d, _files)
     0128:    with open(d.getVar("SCA_DATAMODEL_STORAGE"), "w") as o:
     0129:        o.write(dm_output)
     0130:
 *** 0131:    sca_task_aftermath(d, "oelint", get_fatal_entries(d))
     0132:}
File: '/home/fmartinsons/TAPOS_build/tapos/meta-sca/classes/sca-helper.bbclass', lineno: 229, function: sca_task_aftermath
     0225:    ## Write to final export
     0226:    result_file = os.path.join(d.getVar("T"), sca_conv_export_get_deployname(d, tool))
     0227:    d.setVar("SCA_RESULT_FILE", result_file)
     0228:    with open(result_file, "w") as o:
 *** 0229:        o.write(sca_conv_to_export(d))
     0230:
     0231:    ## Evaluate
     0232:    _warnings = get_warnings_from_result(d)
     0233:    _errors = get_errors_from_result(d)
File: '/home/fmartinsons/TAPOS_build/tapos/meta-sca/classes/sca-conv-to-export.bbclass', lineno: 23, function: sca_conv_to_export
     0019:        func = "sca_conv_dm_{}".format(item)
     0020:        if func in locals().keys():
     0021:            return locals()[func](d)
     0022:        elif func in globals().keys():
 *** 0023:            return globals()[func](d)
     0024:        else:
     0025:            raise NotImplementedError()
     0026:    except bb.parse.ParseError as e:
     0027:        bb.warn(str(e))
File: '/home/fmartinsons/TAPOS_build/tapos/meta-sca/classes/sca-conv-dm-checkstyle.bbclass', lineno: 60, function: sca_conv_dm_checkstyle
     0056:
     0057:    try:
     0058:        return checkstyle_prettify(d, top).decode("utf-8")
     0059:    except:
 *** 0060:        return checkstyle_prettify(d, top)
File: '/home/fmartinsons/TAPOS_build/tapos/meta-sca/classes/sca-conv-dm-checkstyle.bbclass', lineno: 11, function: checkstyle_prettify
     0007:    from xml.etree.ElementTree import Element, SubElement, Comment, tostring
     0008:    from xml.etree import ElementTree
     0009:    from xml.dom import minidom
     0010:    rough_string = ElementTree.tostring(elem, 'utf-8')
 *** 0011:    reparsed = minidom.parseString(rough_string)
     0012:    return reparsed.toprettyxml(indent="  ")
     0013:
     0014:def sca_conv_dm_checkstyle(d):
     0015:    from xml.etree.ElementTree import Element, SubElement, Comment, tostring
File: '/usr/lib/python3.7/xml/dom/minidom.py', lineno: 1968, function: parseString
     1964:def parseString(string, parser=None):
     1965:    """Parse a file into a DOM from a string."""
     1966:    if parser is None:
     1967:        from xml.dom import expatbuilder
 *** 1968:        return expatbuilder.parseString(string)
     1969:    else:
     1970:        from xml.dom import pulldom
     1971:        return _do_pulldom_parse(pulldom.parseString, (string,),
     1972:                                 {'parser': parser})
File: '/usr/lib/python3.7/xml/dom/expatbuilder.py', lineno: 925, function: parseString
     0921:    if namespaces:
     0922:        builder = ExpatBuilderNS()
     0923:    else:
     0924:        builder = ExpatBuilder()
 *** 0925:    return builder.parseString(string)
     0926:
     0927:
     0928:def parseFragment(file, context, namespaces=True):
     0929:    """Parse a fragment of a document, given the context from which it
File: '/usr/lib/python3.7/xml/dom/expatbuilder.py', lineno: 223, function: parseString
     0219:    def parseString(self, string):
     0220:        """Parse a document from a string, returning the document node."""
     0221:        parser = self.getParser()
     0222:        try:
 *** 0223:            parser.Parse(string, True)
     0224:            self._setup_subset(string)
     0225:        except ParseEscape:
     0226:            pass
     0227:        doc = self.document
Exception: xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 388

ERROR: sfxtpm-v1.0.0+gitAUTOINC+07acc0811f-r0 do_sca_oelint_core: not well-formed (invalid token): line 1, column 388
ERROR: sfxtpm-v1.0.0+gitAUTOINC+07acc0811f-r0 do_sca_oelint_core: Function failed: do_sca_oelint_core
ERROR: Logfile of failure stored in: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/aimb231-tapos-linux/sfxtpm/v1.0.0+gitAUTOINC+07acc0811f-r0/temp/log.do_sca_oelint_core.25885
ERROR: Task (/home/fmartinsons/TAPOS_build/tapos/meta-sigfox/meta-tapos/recipes-tapos/infra/sfxtpm_v1.0.0.bb:do_sca_oelint_core) failed with exit code '1

I added traces to print rough_string variable in sca-conv-dm-checkstyle.bbclass and it come from the following malformed XML:

b'<checkstyle version="4.3"><file name="/home/fmartinsons/TAPOS_build/build-tapos/tmp/deploy/images/aimb231/sca/sources/sfxtpm/recipes-tapos/infra/sfxtpm_v1.0.0.bb"><error column="1" line="1" message="[Package:sfxtpm Tool:oelint] Variable \'LICENSE\' should be set" severity="error" source="oelint.oelint_var_mandatoryvar" /><error column="1" line="1" message="[Package:sfxtpm Tool:oelint] Variable \'SECTION\' should be set" severity="error" source="oelint.oelint_var_mandatoryvar" /><error column="1" line="15" message="[Package:sfxtpm Tool:oelint] \'\x1b\' is included multiple times" severity="warning" source="oelint.oelint_var_multiinherit" /><error column="1" line="7" message="[Package:sfxtpm Tool:oelint] DEPENDS should only be appended, not overwritten" severity="error" source="oelint.oelint_vars_dependsappend" /><error column="1" line="40" message="[Package:sfxtpm Tool:oelint] Assignment should be \'VAR += &quot;foo&quot;\' not \'VAR += &quot; foo&quot;\'" severity="error" source="oelint.oelint_vars_inconspaces" /><error column="1" line="26" message="[Package:sfxtpm Tool:oelint] \'SRC_URI\' should be placed before \'inherit\'" severity="warning" source="oelint.oelint_var_order" /></file></checkstyle>

The following part of the XML is the culprit:

<error column="1" line="15" message="[Package:sfxtpm Tool:oelint] \'\x1b\' is included multiple times" severity="warning" source="oelint.oelint_var_multiinherit" />

[priv-kweihmann]

Thanks for pointing this out - definitely a bug, but more in the underlying tool. Will try to fix it there. BTW I never seen multiline inherit, is this something that you commonly use?

[fmartinsons]

Nope, it is pretty recent within our system. Some of our recipes have a lot of inherit so we decided to split this long line (as it is done for example for DEPENDS or RDEPENDS except they have quotes around their value, which inherit have not).

[priv-kweihmann]

I'll work on that as soon as I can - in the meantime you could add oelint.oelint_var_multiinherit to SCA_OELINT_SUPPRESS locally or globally to avoid the issue as a temporary fix

[priv-kweihmann]

That was easier than I thought - Fixed by a new version of the tool

[fmartinsons]

Great ! Thank you.