scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 215 forks source link

Enable test_scrapy_spider() test as it seems to be working now #387

Closed suzil closed 4 years ago

suzil commented 4 years ago

It seems this is due to starting multiple reactors (one in test_scrapy_spider() and another in StrategyWorker). I think the proper solution here would be to use something like a CrawlerProcess and to not execute reactor.run() inside the main code but instead either let the user run it or to use the wrapper like CrawlerProcess. Then we can share a single reactor among different tests (and then could use pytest-twisted). I'm not sure though.