-
My Ubuntu server has 4-core CPU and 8GB RAM. In `scrapyd.conf`, I set the following:
```
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 500
dbs_dir = dbs
max_…
-
**Describe the bug**
Do to this [change](https://github.com/scrapy/scrapyd/commit/3c7a0fc00a3bc62fb32836e76b446454947123fe) in v1.5.0 the regex here (https://github.com/my8100/scrapydweb/blob/8de7ede…
giido updated
1 month ago
-
[scrapyd](https://scrapyd.readthedocs.io/) has scheduling, while this project immediately starts running when a spider is scheduled.
The idea is to start [suspended Kubernetes jobs](https://kubernete…
-
**Describe the bug**
Running scrapyd instances across multiple servers at the moment requires either a complicated tunneling setup (like Wireguard) or directly exposing the scrapyd HTTP interface to …
-
**High Memory Usage with ScrapydWeb**
I've observed an issue where ScrapydWeb causes excessive memory usage when running alongside Scrapyd. On an EC2 instance with 8GB of RAM and 2 vCPUs, Scrapyd a…
-
it would be really good if there would be a simple way to restart the `scrapyd` service after the configuration file `scrapyd.conf` is changed, or even in any circumstance *without killing the current…
-
docker-compose.yml
```
scrapyd:
image: vimagick/scrapyd
ports:
- "6800:6800"
volumes:
- ./data:/var/lib/scrapyd
- /usr/local/lib/python2.7/dist-packages
restart: always
`…
-
Scrapyd has a basic web interface. It would be useful to have something here too, for easy access.
Granted, you can already inspect everything with `docker` or `kubectl`, but a basic web interface st…
-
Scrapyd has configuration options [`jobs_to_keep`](https://scrapyd.readthedocs.io/en/latest/config.html#jobs-to-keep) and [`finished_to_keep`](https://scrapyd.readthedocs.io/en/latest/config.html#fini…
-
We already have `listprojects`, `listspiders`, `schedule`, `addversion`.
https://scrapyd.readthedocs.io/en/stable/api.html
Idempotent:
* [ ] daemonstatus
* [ ] listversions
* [ ] listjobs
…