-
I have started to implement a custom job queue so i can use a shared job queue with Postgre.
I tried to use this setting SPIDER_QUEUE_CLASS but i found out that its missing from Scrapyd. Is it possib…
-
## 廖祥森
#### 本周工作
- xposed：按照之前讨论的策略进行sd卡的清理，可以work，清理速度也蛮快的，之前速度太慢和那个号的数据太多可能有很大关系，已经部署到机子上去
- 课程任务：完成了软件工程研究导引的论文，自然语言处理的第一次作业，分布式算法的读书报告
#### 下周工作
- 龚亿通信那边的事不知道要不要开始开发，总之先了解一下sinosteel，考虑一下…
-
First mentioned in incomplete PR #186
-
If one uploads a project with default version schema which is timestamp using scrapyd-client, and then upload a new version of same project but with GIT version which is specified by ```version = GIT`…
-
Hi
I run scrapyd with the `--rundir` option. (version 1.0.1)
I have the following issue.
- I deploy an egg to scrapyd
- egg and project are present (I can see them in `listprojects.json`)
- I then r…
-
Hello,
I am running about 200 spiders every day at the same time. It's the same spider, but with just different parameters, hence it's being ran 200 times. My Scrapyd config has set 8 jobs on a sin…
-
Hi
I've been experimenting a little with scrapyd on docker, and done the following:
- in the config file, i specified different directory for eggs
`eggs_dir = /src/eggs`
- in dockerfile, …
-
**Describe the bug**
In some situations which I haven't narrowed down yet, the jobs are being killed. It's not clear why, which makes it difficult to fix the problem
**To Reproduce**
Steps to r…
-
Hello,
in my Scrapy spider, I specify the name of the log file as follows:
```
custom_settings = {
'LOG_FILE': f'{datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d_%H%M%S")}_{projec…
-
Hi,
I just start using Scrapy, found this error and don't know how to fix this.
Python 3.6.5 :: Anaconda custom (x86_64)
```
Traceback (most recent call last):
File "/Users/homerun/anac…