scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
530 stars 96 forks source link

Periodic item count monitor #402

Closed ogiaquino closed 1 year ago

ogiaquino commented 1 year ago

Closes #324

Add a monitor suite to periodically monitor increase in item_scraped_count. Increase in item count can be set by adding SPIDERMON_ITEM_COUNT_INCREASE in project settings.

The value for SPIDERMON_ITEM_COUNT_INCREASE can be integer to check for x amount increase in items. The value for SPIDERMON_ITEM_COUNT_INCREASE can be float to check in percentage increase of items.

Below is a sample to enable this monitor suite:

SPIDERMON_ITEM_COUNT_INCREASE = 100

SPIDERMON_PERIODIC_MONITORS = {
    'spidermon.contrib.scrapy.monitors.PeriodicItemCountMonitorSuite': 2,
}
codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 86.95% and project coverage change: +0.05 :tada:

Comparison is base (8251a0f) 78.27% compared to head (444471d) 78.32%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #402 +/- ## ========================================== + Coverage 78.27% 78.32% +0.05% ========================================== Files 73 73 Lines 3102 3124 +22 Branches 516 520 +4 ========================================== + Hits 2428 2447 +19 - Misses 599 600 +1 - Partials 75 77 +2 ``` | [Impacted Files](https://app.codecov.io/gh/scrapinghub/spidermon/pull/402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) | Coverage Δ | | |---|---|---| | [spidermon/contrib/scrapy/monitors/monitors.py](https://app.codecov.io/gh/scrapinghub/spidermon/pull/402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub#diff-c3BpZGVybW9uL2NvbnRyaWIvc2NyYXB5L21vbml0b3JzL21vbml0b3JzLnB5) | `96.66% <85.00%> (-1.23%)` | :arrow_down: | | [spidermon/contrib/scrapy/monitors/\_\_init\_\_.py](https://app.codecov.io/gh/scrapinghub/spidermon/pull/402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub#diff-c3BpZGVybW9uL2NvbnRyaWIvc2NyYXB5L21vbml0b3JzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [spidermon/contrib/scrapy/monitors/suites.py](https://app.codecov.io/gh/scrapinghub/spidermon/pull/402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub#diff-c3BpZGVybW9uL2NvbnRyaWIvc2NyYXB5L21vbml0b3JzL3N1aXRlcy5weQ==) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.