-
First of all thanks a lot for this nifty crawler. I am really enjoying your api design!
Nevertheless, would it be possible to expose the entire [response object](https://github.com/brendonboshell/s…
-
### Description
Traceback shown for an exception raised in a pipeline while opening a spider contains not enough information regarding the pipeline which has thrown this exception.
### Steps to …
-
It'd be great if we could create custom queues or frontiers and inject them into the crawler as an option or parameter.
What I want to do is to use a database to store what urls I have visited and how…
-
I'd like to see an example of custom classifier that is proven to work with custom data. The reason for the request is my headache when trying to write my own and my efforts simply do not work. My cod…
-
- Some "JavaScript-heavy websites" (e.g. https://tripadvisor.com) cannot be scraped by using just Scrapy.
> Can you check why our Beautiful Soup template fails on [tripadvisor.com](https://tripadvi…
-
Hello,
Thanks for the nice robust crawler. But I got ETIMEDOUT error sometimes. Besides, some http request may stay in pending state and the request died after 5 minutes.
Thanks
fanzijian
-
```
What steps will reproduce the problem?
1. Set proxy settings in CrawlConfig
2. Add BasicAuthInfo to CrawlConfig
3. Try to crawl a site with basic authentication
What is the expected output? What …
-
```
What steps will reproduce the problem?
1. Set proxy settings in CrawlConfig
2. Add BasicAuthInfo to CrawlConfig
3. Try to crawl a site with basic authentication
What is the expected output? What …
-
```
What steps will reproduce the problem?
1. Set proxy settings in CrawlConfig
2. Add BasicAuthInfo to CrawlConfig
3. Try to crawl a site with basic authentication
What is the expected output? What …
-
```
What steps will reproduce the problem?
1.
SLES 11.3 with slightly patched 3.16 kernel
Linux memcached9 3.16.3-4.1.100-default #1 SMP Thu Sep 18 06:32:16 UTC 2014
(d2bbe7f) x86_64 x86_64 x86_64 GN…