-
Scrapyd has a basic web interface. It would be useful to have something here too, for easy access.
Granted, you can already inspect everything with `docker` or `kubectl`, but a basic web interface st…
-
We already have `listprojects`, `listspiders`, `schedule`, `addversion`.
https://scrapyd.readthedocs.io/en/stable/api.html
Idempotent:
* [ ] daemonstatus
* [ ] listversions
* [ ] listjobs
…
-
The default Launcher stores `self.processes` as a dict in memory, so when Scrapyd restarts, it loses track of the processes. The processes are stored as `ScrapyProcessProtocol` instances.
A new Lau…
-
i got error message like this when deployed scrapy project to scrapyd, even when scrapy.c…
-
linux：HTTPConnectionPool(host='192.168.0.24', port=6801): Max retries exceeded with url: /listprojects.json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connectio…
-
Regarding distributed setup, this is what I propose. For this setup, we will need scrapyd, rabbitmq, and a distributed file system (HDFS/seaweedfs)
(1) Adding nodes: whatever node we wanna add, we …
-
My team is working on a set of scrapy spiders which we want to deploy to a scrapyd server. Our scrapyd server is configured to use an oauth2 proxy to authenticate traffic.
On all of our requests to o…
-
1. 直接在web管理界面添加scrapyd主机，并且支持有https连接的scrapyd主机
2. 添加中文语言支持
3. 能够配置各种请求的超时时间与重试次数
-
**Describe the bug**
If the ScrapyD Server(s) are running on a remote host (on same VPN) and the ScrapyDWeb is running on a seperate node then the links to Logs and Items become broken by design.
…
-
I want to set priority for my spider.
I done when set on scrapyd:
curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d priority=1
How can i do like that on python-…