Open CarterPape opened 3 weeks ago
Based on how the Scrapy project tests pipelines that return Deferred
s, the callback for the Deferred
should return the item. That suggests that the next pipeline is not invoked until the returned Deferred
finishes.
Also: Duh, the process_item
method is expected to take an item as an argument — no mention in the documentation of accepting a Deferred
.
There is an opportunity to replace the sole dependency on
requests
initem_emailer.py
withscrapy.mail
.As documented in an issue on the Scrapy project, my previous foibles with
scrapy.mail
were the result of failing to see thatscrapy.mail.MailSender().send
returns atwisted.internet.deferred.Deferred
(because this behavior is not documented).An item pipeline can return a
Deferred
, though I don't see documented what happens when this is the case. Does theDeferred
get immediately passed to the next item pipeline? Is the next pipeline called only after theDeferred
finishes? Either way, how is theitem
passed along to the next pipeline?Once I test this and find out, I could add both of these items to the Scrapy documentation.