scrapyrt does not work with scrapy-2.10.0

[michaelbrunnbauer]

From the release notes: "Passing a spider argument to the spider_is_idle(), crawl() and download() methods of scrapy.core.engine.ExecutionEngine, deprecated in Scrapy 2.6, is no longer supported"

Error caught on signal handler: <bound method CrawlManager.spider_idle of <scrapyrt.core.CrawlManager object at 0x7f9e43694f40>>
Traceback (most recent call last):
  File "/home/scrapy/scrapy/lib/python3.8/site-packages/scrapy/utils/signal.py", line 43, in send_catch_log
    response = robustApply(
  File "/home/scrapy/scrapy/lib/python3.8/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/scrapy/scrapy/lib/python3.8/site-packages/scrapyrt/core.py", line 181, in spider_idle
    spider.crawler.engine.crawl(self.request, spider)
TypeError: crawl() takes 2 positional arguments but 3 were given

[pawelmhm]

ok thanks for reporting, will check that.

[pawelmhm]

It looks like the spider argument was removed as part of cleanup here https://github.com/scrapy/scrapy/pull/5090

It was being passed in the past at a time when scrapy could run multiple spiders in one single crawl. We are not relying on having several spiders in single crawl, because we use separate crawl manager and crawler per HTTP request from user.

https://github.com/scrapinghub/scrapyrt/blob/404b7b781c57627add70457e43dc1473d8b83b77/scrapyrt/core.py#L72

https://github.com/scrapinghub/scrapyrt/blob/404b7b781c57627add70457e43dc1473d8b83b77/scrapyrt/resources.py#L255

This means it should be safe to simply remove this argument from a call here. And it could fix the problem. I'll check and test that, unless some volunteers for PR with unit tests. https://github.com/scrapinghub/scrapyrt/blob/404b7b781c57627add70457e43dc1473d8b83b77/scrapyrt/core.py#L181

[pawelmhm]

please test with recent master, I'll make a new release to pypi with this fix this week

[michaelbrunnbauer]

I don't know how to install master but making the one line change from core.py works for me.

[pawelmhm]

ok no problem, new version released now and can be installed from pypi

[michaelbrunnbauer]

The new release also works - thank you!