-
I am having a use case scenario where we are running new job for each task on same domain
I am handling this using `scrapyd` and `schedule.json` API
like
```
curl http://localhost:6800/schedule.json…
-
I'm using `RabbitMQ` with `Scrapy`and when the spider finished, the Job on `Scrapyd` stay on Running status.
Can someone help me to fix that ?
```
2016-10-06 14:05:27 [scrapy] INFO: Scrapy 1.2.0 star…
-
I really like your book, but I have a question: Are there any possibilities to self-host scraping hub? A lot of people have their own infrastructure and so it would be nice to use it. Could you recomm…
-
Inside webservice.py -> lListJobs function:
In the newest code in the repository:
```
running = [{"id": s.job, "spider": s.spider,
"start_time": s.start_time.isoformat(' ')} for s in spid…
-
The pypi scrapyd version does not contain component scrapyd.webservice.DaemonStatus.
-
I was naively getting an argument from the command line and it was ok, but when I moved the project to scrapyd I noticed the arguments were being urlencoded when sent by python-requests. I needed to u…
-
Scrapy 1.0 allows us to run full crawler instances within a process thanks to its internal API.
- Docs at http://doc.scrapy.org/en/0.24/topics/practices.html#running-multiple-spiders-in-the-same-proce…
-
Hi, thanks for the very helpful API wrapper.
I can't find API for send custom argument to the spider, this feature supports in this version?
scrapyd docs: https://scrapyd.readthedocs.io/en/latest/api…
-
Server response (200):
{"status": "error", "message": "NameError: name 'TRUE' is not defined", "node_name": "ubuntu"}
What is the reason?
-
Per [the docs](http://scrapyd.readthedocs.io/en/latest/install.html#installing-scrapyd-in-ubuntu) I'm trying
```
apt-get install scrapyd
```
in ubuntu 14.04 but get this error:
> E: Unable to loca…