search

ScrapeOps / scrapeops-scrapy-sdk

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
https://scrapeops.io/
BSD 3-Clause "New" or "Revised" License
37 stars 9 forks source link

KeyError: `download_latency` #3

Closed janantala closed 2 years ago

janantala commented 2 years ago

Hello, I am getting a lot of Sentry errors after adding Srapeops SDK

Error caught on signal handler: <bound method ScrapeOpsMonitor.log_response_middleware of <scrapeops_scrapy.extension.ScrapeOpsMonitor object at 0x7fabbae0f580>>

KeyError download_latency

scrapeops_scrapy/stats/logger.py in generate_response_stats at line 80

validation = request_response_object.get_validation_test()
geo = request_response_object.get_geo()
custom_tag = request_response_object.get_custom_tag()
custom_signal = 'none'
reslen = len(response_httprepr(response))
total_latency = request.meta['download_latency']

Do you have any idea why?

kerins18 commented 2 years ago

Hey @janantala

That's for letting us know. There must be some edge case with your spider that means its requests don't have a download_latency.

I've just updated the SDK to fix this, so you shouldn't get those errors again. You just need to update the SDK on your machine.

pip install --upgrade scrapeops-scrapy

If you have any feedback on the dashboard, functionality, etc. then we would to hear it. We are working with a number of beta users to refine the functionality, so if it is missing something that you think would be useful for your monitoring needs then please us know. You can email me at info@scrapeops.io.

Ian