scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
528 stars 96 forks source link

Add `JobsComparisonMonitor` #323

Closed tcurvelo closed 1 year ago

tcurvelo commented 2 years ago

closes #301

It adds the JobComparisonMonitor, for checking for drops in scraped item count compared to previous jobs, when running jobs in Scrapy Cloud.


TODO:

codecov[bot] commented 2 years ago

Codecov Report

Merging #323 (d3f5a7f) into master (06d70af) will increase coverage by 0.58%. The diff coverage is 88.23%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #323      +/-   ##
==========================================
+ Coverage   74.20%   74.79%   +0.58%     
==========================================
  Files          68       68              
  Lines        3032     3067      +35     
  Branches      462      469       +7     
==========================================
+ Hits         2250     2294      +44     
+ Misses        719      705      -14     
- Partials       63       68       +5     
Impacted Files Coverage Δ
spidermon/utils/zyte.py 84.00% <84.00%> (ø)
spidermon/contrib/scrapy/monitors.py 97.57% <94.11%> (-0.69%) :arrow_down:
spidermon/contrib/scrapy/extensions.py 85.26% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 06d70af...d3f5a7f. Read the comment docs.

curita commented 2 years ago

@rennerocha hi! Bothering you again 🙈 This PR is currently stalled, but the related ticket is assigned for the next milestone so I imagine we'd like to move forward with it. Do you think it would be ok to assign someone from Zyte to work on it? If you'd like to continue with it instead no worries, we can work on something else, let us know what you'd like to do!