-
1、希望可以增加下爬虫的日志监控和报警，方便及时的对爬虫进行异常修复和管理，例如：网络中断、页面改变，反爬等，都会导致爬虫异常，这些信息应该及时的反馈到gerapy平台上供管理员查看；
2、项目部署时，是将项目中的所有爬虫部署到某一台服务器上，是否可以将支持只部署其中的某几只爬虫呢？
-
I want to call the /schedule.json interface, return a jobid or pass in a jobid, and then when scheduling a job, I want to be able to get this ID in the pipelines, what do I need to do?
-
(Coming from here: https://bugs.python.org/issue28630)
I'm trying to prepare an egg file without source code.
I had used `python setup.py bdist_egg --exclude-source-files`, which does not work wit…
-
At the end of each phase of data processing we should evaluate whether it ended well, there is something suspicious or this particular phase failed.
For `collect` phase define criteria that will
a…
-
When adding you module and trying to deploy to Heroku with Python 3.6.4 I get the following error:
```
remote: Collecting distribute (from scrapy-heroku==0.7.1->-r /tmp/build_a2015ad4e03aa9…
-
For anyone receives `NoneType' object has no attribute 'group`, if you are currently using `googletrans==3.0.0,` please switch to `googletrans==3.1.0a0` for the temporary fix.
Related Issue: #234
-
At the end of each phase of data processing we should evaluate whether it ended well, there is something suspicious or this particular phase failed.
For `process` phase define criteria that will
a…
-
How to solve this..
tried to install it on a Mac and a clean Ubuntu server same problem...
-
### Description
A spider inherits SitemapSpider parcing sites sitemaps, starting from `robots.txt`, has `JOBDIR` set.
I run it as a `CentOS 8.x` service with a unit file defined and it runs …
-
See https://github.com/scrapy/scrapyd-client#versioning
This means (if we have a use case) that we can send to Kingfisher Process the code version, to know which version of a spider was run (for tr…