zytedata / zyte-spider-templates

Spider templates for automatic crawlers.
BSD 3-Clause "New" or "Revised" License
22 stars 4 forks source link

CrawlingLogsMiddleware truncating logs #67

Open gutsytechster opened 4 weeks ago

gutsytechster commented 4 weeks ago

The middlewares when included in the SPIDER_MIDDLEWARES setting causes error from the callbacks to be truncated and shown like below image

After I remove this middleware, the logs are shown with full traceback, showing which line cause the specific errors, which is the intentional behaviour.

Gallaecio commented 4 weeks ago

For the record, this is most likely because of the middleware casting the results iterable to a list instead of iterating it.