jmaassen commented 7 months ago

Mention scraper fixed

Fixes #1085 #1084

Changes proposed in this pull request:

Citation scraper should use try-with-resources when performing HTTP requests. Otherwise, we run the risk of a excessive number of threads being created when a large number of citations are found leading to out-of-memory errors.
Same is true for other scrapers, although the issue was not triggered there.
When scraping citations, the result must be deduplicated before pushing it to the database. Otherwise, the entire operation fails, and no citations are added at all.
Use loggers in the scrapers to allow for proper logging and debugging
Some additional refactoring of scrapers to cleanup of issues detected by sonarcloud

How to test:

Remove any existing data with docker compose down --volumes
Build with docker compose build
Start an empty RSD with scrapers docker compose up --scale data-generation=0 --scale scrapers=1
Sign in, create a test software item, and add 10.1021/ja026939x as reference paper
Wait for the CitationScraper to run or docker compose exec scrapers /opt/java/openjdk/bin/java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainCitations manually. This takes about a minute...
Check if the reference paper now has 2562 "referenced by" entries

PR Checklist:

jmaassen commented 7 months ago

In further testing I still see a database error pop up whenever this scraper runs:

ERROR:  duplicate key value violates unique constraint "mention_external_id_source_key"

I converted the PR to draft and will investigate further.

sonarcloud[bot] commented 7 months ago

Quality Gate passed for 'rsd-database'

Kudos, no new issues were introduced!

sonarcloud[bot] commented 7 months ago

Failed conditions

idea Catch issues before they fail your Quality Gate with our IDE extension SonarLint