scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
524 stars 94 forks source link

Add handling for datetime aware start time in PeriodicExecutionTimeMonitor #423

Closed further-reading closed 2 months ago

further-reading commented 9 months ago

In the current version of scrapy, the code below breaks due to spider start time being timezone aware https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/monitors/monitors.py#L497

now = datetime.datetime.utcnow()
start_time = self.data.stats.get("start_time")
duration = now - start_time

It currently throws the following error:

TypeError: can't subtract offset-native and offset-aware datetimes

Backwards compatible Fix: add try/except to add utc timezone to now timestamp when error thrown.

curita commented 2 months ago

438 took care of this issue.