-
During tests I observed a couple of times that a fetch failed due to 0 bytes being returned from the server. Since it was not deterministic, a simple "retry" could probably work, but there is currentl…
-
```
What steps will reproduce the problem?
1. Start a crawler with some web site which contains on some of pages string
like "spłaty pożyczkiCzy"
2. Print out html text in WebCrawler.visit:
System.ou…
-
I noticed new 'empty' session objects get saved. As sessions are typically saved for an hour these can soon mount up, especially from clients which don't retain cookies (ie bots / webcrawlers etc).
…
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …
-
@EvanHahn do you have plans to support pattern-matching rules (for web crawlers) like:
- Disallow: /private*/
- Disallow: /_?id=_
- Disallow: /image.php?*
- Disallow: /*.xls$
More information about p…
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …
-
```
Doing this will allow usage of for example Spring instantiated objects inside
Crawler4J.
I attach zip with 2 classes, that can be used to patch crawler4J project.
```
Original issue reported …