search

scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
526 stars 94 forks source link

Update `BaseStatMonitor`'s `test_stat_monitor` to work with Scrapy Cloud #378

Closed mrwbarg closed 1 year ago

mrwbarg commented 1 year ago

Fixes #374.

When reading threshold values from the settings, the BaseStatMonitor used the settings.get() function. This works fine if the value is set in the project's settings as the data type is preserved when reading it. However, when getting the settings from a Scrapy Cloud project, using the bare get() function causes the value to be read as a string, which break comparisions.

The monitor was updated to use by default the getfloat function. A variable was added which allows setting the threshold data type (either int or float) along with a property that picks the correct get function depending on the data type.

Two new unit tests were added to test for the cases where the values are actual strings.

Despite there being other monitors that inherit from BaseStatMonitor, no further changes were necessary as they'll benefit from the default getfloat() function.

codecov[bot] commented 1 year ago

Codecov Report

Base: 74.85% // Head: 74.89% // Increases project coverage by +0.03% :tada:

Coverage data is based on head (5432ad6) compared to base (8846c75). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #378 +/- ## ========================================== + Coverage 74.85% 74.89% +0.03% ========================================== Files 73 73 Lines 3158 3163 +5 Branches 373 373 ========================================== + Hits 2364 2369 +5 Misses 727 727 Partials 67 67 ``` | [Impacted Files](https://codecov.io/gh/scrapinghub/spidermon/pull/378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub) | Coverage Δ | | |---|---|---| | [spidermon/contrib/scrapy/monitors.py](https://codecov.io/gh/scrapinghub/spidermon/pull/378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub#diff-c3BpZGVybW9uL2NvbnRyaWIvc2NyYXB5L21vbml0b3JzLnB5) | `96.72% <100.00%> (+0.07%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapinghub)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.