Address Content Extraction Issues in Thread Using Newspaper Library

[KadeMorton]

Describe the bug Since adjustments were made to enhance the comprehensiveness of text extraction in Thread, utilizing the Newspaper library, new issues have arisen, including text duplication and incorrect ordering of content. This affects the data extracted from websites.

To Reproduce

Environment set-up:

• Use a public instance: Thread is hosted live via Arachne
  • Newspaper version used is this PR: jecarr/newspaper#1
• Run locally: if running a local version, you can replicate this issue by using the previously-stated Newspaper version.
  • Alternatively you can use the original Newspaper version (or look into other forks) and gauge if this issue exists and/or the missing-text issue.

Steps to reproduce the behaviour:

1. Go to the Thread homepage
2. Click Enter Reports
3. Fill in the associated information. Any URL will do
4. Wait for the report to be processed
5. Once the report is in News Review, click Analyse
6. Look through the text Thread presents and compare it to the original website

Expected behavior Text to not be duplicated, and text to be in order. It's understandable if some text is not brought over given the vast variety of websites, but what is brought over should not be duplicated and in order.

Thread details (please complete the following information):

• config.yml changes
• Tagged-version used or output of git rev-parse --short HEAD

Desktop (please complete the following information):

• OS: Windows 11
• Browser Firefox
• Version 125.0.1

Acceptance Criteria

• Assess and diagnose the root cause of text duplication and disordered content in the output from the Newspaper library.
• Develop and implement a solution that ensures text is extracted in the correct order and without unnecessary duplication.
• Validate that the solution does not reintroduce previous issues of missing text chunks as much as possible.
• Ensure that the implemented solution is efficient and does not significantly degrade performance or increase processing times.
• Document the changes made to the Newspaper configuration or code, providing clear reasoning for each adjustment for future reference.

[cvallance]

Found the issue with things getting out of order, easy fix. Was in the web_svc.py code and not newspaper. Will post a MR shortly.

@KadeMorton Can you give me an example URL for a page that duplicates data? I can't reproduce.

[KadeMorton]

We've chatted over email, but in case anyone is looking over tickets I don't want it to look like we ignored you! https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-242a is the URL that I passed over.

There are not a lot of duplicated sentences for this URL when you run it through Thread, but there are a couple and I've checked on the original source page, the duplicated text is definitely only there once. There are some reports where the dupes are quite prevalent, some where its just a few sentences, and then some where its none.

You've indicated you're well on your way and we can keep chatting over Slack.