Infomedia cleaning edge cases

[jankounchained]

if you want to be extra careful, then you control for two edge cases:

• add a multiline flag in case there is a long html tag between mulitple lines
• replace html tags with a single shitespace in case that this happens: word<br /></p><p id=\"p1\">word
• take care of excess whitespace with another function

so something like this:

def remove_html_tags(text: str) -> str:
    """Remove HTML tags from a string."""
    html_tag_pattern = re.compile('<.*?>', flags=re.MULTILINE)
    clean_text = re.sub(html_tag_pattern, " ", text)
    return clean_text

def remove_whitespace(text: str) -> str:
    """remove excess whitespace from text fields"""
    pat_ws = re.compile(pattern=r"\s\s+", flags=re.MULTILINE)
    clean_text = re.sub(pat_ws, " ", text)
    return clean_text

[KennethEnevoldsen]

@jankounchained do you plan to fix this or should @TTTTao725 ?

[TTTTao725]

Sure, no problem! I'll be working on it tomorrow, I‘m volunteering for a user test today, and tomorrow I might be wearing a brace 😹

[KennethEnevoldsen]

@TTTTao725 is this issue resolved?

[TTTTao725]

yes, please check out this PR: https://github.com/centre-for-humanities-computing/danish-foundation-models/pull/257