-
I am extending this module of heritrix `org.archive.modules.fetcher.FetchHTTP` and overriding the innerProcess method to make a headless browser get the content instead of the builtin heritrix http re…
-
Hi,
I'm writing a paper and would like to cite heritrix. What/Who/... do I need to reference? I found the following:
- [Mohr, G., Stack, M., Rnitovic, I., Avery, D., & Kimpton, M. (2004, July). An…
-
Hello,
I am trying to understand crawler-beans.cxml usage in Heritrix but can't find any best practice articles or detailed manual anywhere. There are a few documentation floating on the net and so…
-
Hi!
I've crawling with Heritrix for 5 hours and I've noticed that the `state` directory size is bigger even that my downloaded WARCs. My configuration, briefly:
- Seeds: ~1200 domains
- Downloadi…
-
![image](https://user-images.githubusercontent.com/38576501/138016636-6e12d82e-2b8c-46df-910a-cd3c5a3b96c8.png)
-
Hi! First of all, I'm sorry if this question has been covered in documentation or previous issues, but I have not been able to find information about this.
I have a use case where I need to disabl…
-
I am aware of the documentation on "Common Heritrix Use Cases" in the wiki to mirror only html files or exclude rich media. Still, I don't get my job to work that should simply not download and / or w…
-
What are some strategies to reduce disk usage? I am crawling a specific forum starting at its home page and re-starting crawls through the REST API once per hour. I had hoped this would reduce the amo…
-
I have noticed a lot of bad requests from archive.org's crawler on our sites using Google Tag Manager. For instance:
```
/in.tag/11/10/2024/gtm.start/
/mouseup.dismiss/11/10/2024/gtm.start
/moused…
-
In my crawls, I get several REJECTS in `SchemeNotInSetDecideRule` based on the data uri scheme (`data:`). How can I add this in the corresponding bean?
Thanks for a hint.