Christovis commented 2 years ago

This PR focuses on the scraping of mailing archives from W3C (bigbang/w3c.py) and Listserv (bigbang/listserv.py). As they have multiple methods in common, they can be abstracted out into a new bigbang/abstract.py file containing AbstractMessageParser, AbstractList, and AbstractArchive.

As a consequence:

our scraping of W3C mailing archives becomes more sophisticated (e.g., now we can scrape the whole archive, filter it by time ranges and message content) and the
scraping of different mail archives are built around the same template which safes us from duplicating code

This is part of the conversation in #512.

codecov-commenter commented 2 years ago

Codecov Report

Merging #534 (3ebffc7) into main (2c41b5c) will increase coverage by 0.76%. The diff coverage is 84.04%.

@@            Coverage Diff             @@
##             main     #534      +/-   ##
==========================================
+ Coverage   74.24%   75.00%   +0.76%     
==========================================
  Files          22       27       +5     
  Lines        3075     3305     +230     
==========================================
+ Hits         2283     2479     +196     
- Misses        792      826      +34

Flag	Coverage Δ
unittests	`75.00% <84.04%> (+0.76%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tests/unit/test_bigbang.py	`100.00% <ø> (+2.11%)`	:arrow_up:
bigbang/ingress/mailman.py	`61.61% <66.66%> (+0.19%)`	:arrow_up:
bigbang/ingress/utils.py	`70.51% <70.51%> (ø)`
bigbang/ingress/w3c.py	`74.27% <74.27%> (ø)`
bigbang/bigbang_io.py	`82.11% <84.37%> (-0.29%)`	:arrow_down:
bigbang/ingress/abstract.py	`85.55% <85.55%> (ø)`
bigbang/ingress/listserv.py	`83.28% <87.50%> (ø)`
tests/ingress/test_listserv.py	`90.62% <88.88%> (ø)`
bigbang/analysis/listserv.py	`63.71% <90.90%> (+1.01%)`	:arrow_up:
bigbang/__init__.py	`100.00% <100.00%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2c41b5c...3ebffc7. Read the comment docs.