search

scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
524 stars 94 forks source link

Field names containing "_items" crash field coverage #428

Closed rvandam closed 4 months ago

rvandam commented 6 months ago 
  File "/Users/rvandam/.venv/ADD1D703-D26A-4DD9-BEE9-DA619F3B2F68/lib/python3.11/site-packages/scrapy/utils/defer.py", line 348, in maybeDeferred_coro
    result = f(*args, **kw)
  File "/Users/rvandam/.venv/ADD1D703-D26A-4DD9-BEE9-DA619F3B2F68/lib/python3.11/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/Users/rvandam/.venv/ADD1D703-D26A-4DD9-BEE9-DA619F3B2F68/lib/python3.11/site-packages/spidermon/contrib/scrapy/extensions.py", line 128, in spider_closed
    self._add_field_coverage_to_stats()
  File "/Users/rvandam/.venv/ADD1D703-D26A-4DD9-BEE9-DA619F3B2F68/lib/python3.11/site-packages/spidermon/contrib/scrapy/extensions.py", line 182, in _add_field_coverage_to_stats
    coverage_stats = calculate_field_coverage(stats)
  File "/Users/rvandam/.venv/ADD1D703-D26A-4DD9-BEE9-DA619F3B2F68/lib/python3.11/site-packages/spidermon/utils/field_coverage.py", line 29, in calculate_field_coverage
    item_field_coverage = value / root_field_type_total
TypeError: unsupported operand type(s) for /: 'int' and 'NoneType'

Problem seems to be in line 19 of utils/field_coverage.py -> calculate_field_coverage which makes the assumption that if the field name contains "_items" it must be an internal coverage stat.

Perhaps that line should be if "/_items/" in item_key: ??