scrapinghub / scrapyrt

HTTP API for Scrapy spiders
BSD 3-Clause "New" or "Revised" License
824 stars 161 forks source link

scrapyrt becomes unresponsive #133

Open aanalagile opened 2 years ago

aanalagile commented 2 years ago

I have been using scrapyrt on a live project, it works perfectly fine and fulfils all it's purposes. But sometimes it just stops responding, I run scrapyrt with systemd service which runs scrapyrt -p 3000 command. It accepts the request but doesn't return any response. I'm not sure what is causing the issue, I tried to identify by using sending concurrent requests ,etc. So to get it back working I have to restart the service and then it starts working again. I'm not sure but is it a bug or could be some scenario which causes this behaviour? Is there any way builtin way which can restart the scrapyrt incase it stops responding. Or there is a way by which I can identify if it is broken again , so I can restart it manually.