Using BeautifulSoup4 and html.parser to parse XML FIles.

[35C4n0r]

In order to parse XML documents we will be using BeautifulSoup4 and html.parser. Now we are using this option instead of the Python's built in XML Parser is because at times the XML that has to be parsed is malformed and the html.parser is linient in parsing, whereas the standard library only handle well-formed XML.

There is an issue with this approach:

• Parsing the document as HTML, all the tags were converted to lower case by default, and there is no option to change that.

In order to deal with this:

• We create a mapping of all the tags to another another tags and replace those tags to the mappings.
• We then parse the document normally as HTML.
• We then use the mapping to replace the mapped tags to the original tags.

Example:

<parent>    
    <groupId>org.jboss.seam</groupId>   
    <artifactId>root</artifactId>   
    <url>https://github.com/</url>  
    <Url>https://github.com/35C4n0r</Url>   
</parent>

The Mapping:

{'groupId': 'TAG0', 'parent': 'TAG1', 'url': 'TAG2', 'artifactId': 'TAG3', 'Url': 'TAG4'}

The new XML

<TAG1>  
    <TAG0>org.jboss.seam</TAG0> 
    <TAG3>root</TAG3>   
    <TAG2>https://github.com/</TAG2>    
    <TAG4>https://github.com/35C4n0r</TAG4> 
</TAG1>

After parsing it with BeautifullSoup

<tag2>
    <tag4>org.jboss.seam</tag4>
    <tag1>root</tag1>
    <tag0>https://github.com/</tag0>
    <tag3>https://github.com/35C4n0r</tag3>
</tag2>

After using the map to convert the tags back

<parent>
    <groupId>org.jboss.seam</groupId>
    <artifactId>root</artifactId>
    <url>https://github.com/</url>
    <Url>https://github.com/35C4n0r</Url>
</parent>

[pombredanne]

@35C4n0r Can you elaborate on what issue you are facing with details? Which exact files are a problem to parse?

[35C4n0r]

@35C4n0r Can you elaborate on what issue you are facing with details? Which exact files are a problem to parse?

@pombredanne I was just creating this on the Project Board, accidently converted this to an Issue. Anyways I've added a proper description.

[pombredanne]

@35C4n0r thanks! I was just curious! I sometimes do this too. Now I wonder if we ever care about the XML tag case? Because if so you do not even need to build a mapping at all.

[35C4n0r]

@pombredanne we use the POM which has properties attribute, we depend on tag names to extract different properties, example.