-
## 500 (INTERNAL SERVER ERROR): 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the…
-
Description:
When running Scrapyd with Python 3.11.9, the format of the HTML returned on the Jobs page appears to be standardized in a way that prevents successful parsing of job data. This issue doe…
-
**High Memory Usage with ScrapydWeb**
I've observed an issue where ScrapydWeb causes excessive memory usage when running alongside Scrapyd. On an EC2 instance with 8GB of RAM and 2 vCPUs, Scrapyd a…
-
**Describe the bug**
Do to this [change](https://github.com/scrapy/scrapyd/commit/3c7a0fc00a3bc62fb32836e76b446454947123fe) in v1.5.0 the regex here (https://github.com/my8100/scrapydweb/blob/8de7ede…
giido updated
1 month ago
-
[scrapyd](https://scrapyd.readthedocs.io/) has scheduling, while this project immediately starts running when a spider is scheduled.
The idea is to start [suspended Kubernetes jobs](https://kubernete…
-
it would be really good if there would be a simple way to restart the `scrapyd` service after the configuration file `scrapyd.conf` is changed, or even in any circumstance *without killing the current…
-
**Describe the bug**
Running scrapyd instances across multiple servers at the moment requires either a complicated tunneling setup (like Wireguard) or directly exposing the scrapyd HTTP interface to …
-
docker-compose.yml
```
scrapyd:
image: vimagick/scrapyd
ports:
- "6800:6800"
volumes:
- ./data:/var/lib/scrapyd
- /usr/local/lib/python2.7/dist-packages
restart: always
`…
-
I've been debugging this problem for a while, it's intermittent making it harder to reproduce.
When running some jobs with `scrapy-playwright` the jobs get's abruptly terminated, if you observe th…
-
Scrapyd has configuration options [`jobs_to_keep`](https://scrapyd.readthedocs.io/en/latest/config.html#jobs-to-keep) and [`finished_to_keep`](https://scrapyd.readthedocs.io/en/latest/config.html#fini…