Our pgh_public_schools spider is leaving some HTML in the event description. It would be nice to clean this up by stripping out the HTML. Here are some examples:
In other spiders, we have had some success with this code snippet from StackOverflow. We might want to make this a reusable function somewhere in the project:
from io import StringIO
from html.parser import HTMLParser
class MLStripper(HTMLParser):
def __init__(self):
super().__init__()
self.reset()
self.strict = False
self.convert_charrefs= True
self.text = StringIO()
def handle_data(self, d):
self.text.write(d)
def get_data(self):
return self.text.getvalue()
def strip_tags(html):
s = MLStripper()
s.feed(html)
return s.get_data()
Our
pgh_public_schoolsspider is leaving some HTML in the event description. It would be nice to clean this up by stripping out the HTML. Here are some examples:In other spiders, we have had some success with this code snippet from StackOverflow. We might want to make this a reusable function somewhere in the project: