scrapinghub / scrapy-poet

Page Object pattern for Scrapy
BSD 3-Clause "New" or "Revised" License
119 stars 28 forks source link

Define expected behavior when passing PO as part of cb_kwargs #98

Closed VMRuiz closed 1 year ago

VMRuiz commented 1 year ago

I was working on a project where I manually had to retry some requests like:

def parse(self, response, parameter, po: PoetClass):
   if broken:
       return Request(response.url, cb_kwargs=response.request.cb_kwargs, self.parse)
    # [...]

In this case, the current po is passed to the next invokation of parse so its content is not recreated with the new value of response. I was talking with @Gallaecio about this and while I think this behavior is not wrong, it took me a while to figure out why I was not getting new data in po until I realised my poor decision.

So, I think this should be allowed because there may be a case where someone want to pass and already created Poet Object instead of expecting a new one to be created. Nevertheless, as this is a quite rare case, maybe the library could log a warning to let the user know that he may be doing something wrong in most of the other cases.

Gallaecio commented 1 year ago

@lampssy noticed that this issue no longer applies since https://github.com/scrapinghub/scrapy-poet/pull/113.