-
This is a security-related issue, and I have read https://github.com/internetarchive/heritrix3/blob/master/SECURITY.md. Because it is not specific to heritrix3, and is getting a lot of public attentio…
-
Heritrix version : 3.4.0-20210803
I have added my full configuration file here
https://gist.github.com/naveen17797/b119d62ef3c0a20e656cae5ea56d5821
```
…
-
I'm trying to crawl a company website using the seed www.example.com and Heritrix is either generating no warc file or just empty warcs.
I'm using a simple one line seed of the company website.
If…
-
Redirected links (to the web archive) on the BL website are failing to take users to the latest instance:
https://www.bl.uk/catalogues-and-collections/digital-collections
![image](https://user-image…
-
Hi! I have some questions about `TransclusionDecideRule` which I'd appreciate if someone could answer:
- The option `maxTransHops` with the value 0 means that only text or similar will be downloade…
-
I am trying to crawl a website where I want Heritrix to ignore robots.txt and archive everything. However, despite the fact that I have `robotsPolicyName` set to `ignore`, it's still looking at robots…
-
严重: Can't create bean 'fetchDns'; Can't create bean 'fetchProcessors'; Can't create bean 'crawlController'; Can't create bean 'frontier'; Can't create bean 'seeds'
org.springframework.beans.factory.U…
-
Hi!
I'm crawling with the Disk Space Monitor enabled:
```xml
/
```
Sadly for me, it was triggered, but I noticed that the crawl time didn't sto…
-
### Behaviour
#### Steps to reproduce this issue
1. Create/build an image because we want to test if the changes pass (following https://github.com/docker/build-push-action/blob/master/docs/adva…
-
Hi,
I am trying to crawl some WWW domain from behind a web proxy in a corporate network using Heritrix build 3.3.0-20180727.011238-114 without success. Heritrix just hangs right after the crawl is …