-
Hi, i saw in a changelog that :
> CHANGELOG
> =========
> 0.3.0
> -----
> * Add a PHPUnit extension to keep alive the webserver and the client between tests
>
> 0.2.0
> -----
> * Allow ke…
-
Currently, we use a dedicated redis instance for a crawl job. It takes some amount of scheduling time and overhead (CPU/Memory) to bring its own redis instance. There are two ways to try for this issu…
-
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After
To reproduce:
```$ curl -L -I http://reddit.com```
It should yield a 429 at some point, when trying to hit `https://www.r…
-
Hello, I followed quick start tutorial. Turned on db worker and strategy worker. Then I turned on scrapy using `scrapy crawl general -L INFO -s FRONTERA_SETTINGS=frontier.spider_settings -s SEEDS_SOUR…
-
I've filed a support case with AWS, but just a heads-up, and to save anyone else out there searching for answers...
AWS CloudFront launched a new Cache/Origin Policies feature a few weeks ago. Whil…
-
### What happened?
When attempting to run `Build.build(custom_parser)` I get a `pydantic` error that kills the catalog build, despite following the tutorial and successfully running the parser on t…
-
On a fresh VPS I imported a wordpress website, purchased and installed the plugin.
When I try to run it, the UI says
500 error code returned from server.
Please check your server's error logs or…
-
### Context
From a european point of view cookies are troublesome. Most sítes are forced to ask the user to accept cookies due to the ePrivacy Directive. And we don't want to make browserprofiles f…
-
```
What steps will reproduce the problem?
1. Create a web-page with a malformed URL (or a protocol like mailto:)
2. Run the crawler on said website.
3. Crash and burn at line 89 in WebURL.java - this…
-
I installed master/45995736 on OS X 10.8.5 today. I'm using ruby 1.9.3-p429 via rbenv, and when I try to run the crawler from the "rubycode" directory, I get a gem specification failure:
```
$ ruby m…
irons updated
10 years ago