-
Hello, I'm using scrapyrt to provide an HTTP interface to a big Scrapy project. I'm running a single scrapyrt instance in a Docker container, some spiders require ~60-120 seconds to complete and I've …
-
Hi! thanks for your work on Scrapyrt!
I've discovered that spiders served by Scrapyrt don't save the output in the Spider's / custom_settings / FEEDS. Is it possible to change this behavior and ma…
runa updated
6 months ago
-
Hi Team,
I am trying to override the log settings so that I can dump my logs to a custom file in a custom directory. However, the scrapyRT code never allows a user's custom settings configs to be u…
-
`max_items` like the `max_requests` argument would be really helpful.
-
I am using scrapyrt in Google App Engine. To save the logs in Google Cloud i need to log to stdout but scrapyrt does not support that see:
https://github.com/scrapinghub/scrapyrt/blob/f8ee7b79fcfeaf4…
-
I'm experiencing difficulties in accessing a ScrapyRT service running on specific ports within a Kubernetes pod. My setup includes a Kubernetes cluster with a pod running a Scrapy application, which u…
-
Actually i read scrapyrt documentation and its mention that i need to modify ScrapyRT and need to use websockets or HTTP push notifications.But i don't understand how to do this?...I also try to find …
-
I have been exploring many options on how to keep scrapyrt open and active even after reboot, but I am unsure what is best. I was thinking of using [immortal.run](https://immortal.run/). I have used…
-
(Sorry can't find how to label this)
I hope this is the right place where to ask this.
I created a spider that can scrape a page in an e-commerce site and gather the data on the different items.
…
-
AFAICT it's not possible to override LOG_LEVEL, LOG_FILE, LOG_DIR, etc for spiders because the dict from get_scrapyrt_settings is applied with priority 'cmdline'.
I assume this is due to conflicting …