-
Hi Medialab,
Running sudo bin/deploy_scrapy_spider.sh to install the latest edition of hypher gives me the following error:
Copying config.json from root directory to hyphe_backend/crawler for scra…
-
Just a small issue I've been having with scrapyd. Four of my jobs encountered MySQL errors and then grinded to a halt. Wondering how to handle this? The jobs have been running for over 2 days and ther…
-
-
Hi MediaLab,
I'm trying to setup scholarScape following your suggestions on the front page. Everything worked fine until reaching this line:
twistd -ny scholarScape.tac
First of I thinking that you…
-
Hi,
we were running into problems where we would get the following exception when crawling urls:
```
Traceback (most recent call last):
[...]
field = item.fields[field_name]
exceptions.Attribute…
-
New features and settings:
> ...
> Real support for returning iterators on start_requests() method. The iterator is now consumed during the crawl when the spider is getting idle (r2704)
> ...
This w…
-
Having discovered the bash script bin/runtests.sh, running it;
[system python2.7]
```
archtester scrapy # ebuild scrapy-0.14.2.ebuild clean test
-----------------------------------------------------…
-
changed scrapyd data dir to be ".scrapy/scrapyd" instead of ".scrapy/.scrapy/scrapyd"
```
--- a/scrapyd/script.py
+++ b/scrapyd/script.py
@@ -14,7 +14,7 @@ from scrapyd import get_application
from sc…
-
Previously reported by agtilden on Trac
scrapyd only allows one job to be scheduled per URL invocation. This makes scheduling lots of jobs needlessly time consuming.
I propose adding a file upload o…