Article being double posted

[changhuapeng]

As per title, the bot is double posting certain articles from straitstimes.com

• https://www.reddit.com/r/singapore/comments/7nldsn/waiter_who_had_unprotected_sex_with_secondary/ds2nls9/
• https://www.reddit.com/r/singapore/comments/7nkw8j/fines_for_personal_mobility_devices_to_treble/ds2km3d/
• https://www.reddit.com/r/singapore/comments/7nktm2/man_charged_over_abuse_assault_of_police_officer/ds2k47e/
• https://www.reddit.com/r/singapore/comments/7njeld/track_signalling_fault_recurs_on_eastwest_line_on/ds28jj7/

The Mercury web parser seems to be confused with <div class="row">, an outer div element that encapsulates the whole article to be <p class="row">. The bot then mistakenly post the content in this <p class="row"> assuming it to be the first paragraph before moving on to post the actual scraped paragraphs.

[changhuapeng]

One method is to replace the soup.select("p"), at line 43 of mercury.py, with soup.find_all(lambda tag: tag.name == "p" and not tag.attrs) with the assumption that all legitimate paragraphs are encapsulated with <p> without any classes, id, etc attributes. This means that the mistaken <p class="row"> will then be then omitted by the bot because of the class attribute.

I have tested out this solution with multiple past articles posted by the bot and found it to be generally reliable except for this yahoo article and scmp.com article where certain paragraphs are omitted by the bot because of their <p class="xxxx"> tag.