Stop Iteration Error while using run_in_reactor

[this-is-r-gaurav]

I am trying to Integrate Django With Scrapy and for running Crawler with run_in_reactor instead of wait for as shown in example How to integrate Flask & Scrapy? But This raises error

Unhandled error in EventualResult
Traceback (most recent call last):
  File "/project/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 460, in callback
    self._startRunCallbacks(result)
  File "/project/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/project/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/project/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/project/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
builtins.StopIteration: 

What i want is After Running the function which is decorated by run_in_reactor the user of the program get alert that the function is called successfully. And as the result finished, it doesn't lead to this error

[this-is-r-gaurav]

Solved The Issue Thanks i was passing Crawlrunner object as Parameter, as the function stop. The Object gets destroyed. This leads to Stop Iteration Error I guess

[MarwaKH29]

Hi, i have the same error occuring when i run a spider within a flask app but i don't get a response from the spider i am using flask, crochet(1.12.0), scrapyd, twisted(19.10.0)

[HimavarshaVS]

Solved The Issue Thanks i was passing Crawlrunner object as Parameter, as the function stop. The Object gets destroyed. This leads to Stop Iteration Error I guess

I am using the same method and facing same issue. how did you resolve it. any help would be great

[dzhen19]

Hello my brother I have the same issue.

[this-is-r-gaurav]

The Reactor Code look like this, which call upon GeneralSpider which fetches all the necessary data based on the pk as primary key.

runner = CrawlerRunner()
output = []
@run_in_reactor
def scraped_by_id(pk):
    global runner
    dispatcher.connect(_crawler_result, signal=signals.item_scraped)
    eventual = runner.crawl(GeneralSpider, id=pk)
    dispatcher.connect(_crawler_Stop, signals.engine_stopped)
    return eventual

def _crawler_result(item, response, spider):
    global output_data
    output_data.append(dict(item))

def _crawler_Stop():
    # Perform any operation which is relevant to your application, like notify user
    pass

And my GeneralSpider class look like this. Where Job contains info like url to scrape, fields to scrape, etc.

class GeneralSpider(scrapy.Spider):
    def __init__(self, id, **kwargs):
        self.id = id
        self.custom_settings = {"LOG_LEVEL": "DEBUG"}

        job = Job.objects.get(pk=id)

Hope this helps @MarwaKH29 @HimavarshaVS @dzhen19

[Akkisdiary]

Can you please show the code for def _crawler_Stop() ? And also my imagePipeline and deltafetch middleware are also not working??