-
### Is your proposal related to a problem?
With Wagtail 2.11 internationalisation got a huge update and i am a big fan of it already.
One thing i am missing are the entries in the sitemap.xml …
-
```
ZAP currently uses DirBuster 0.12
We should upgrade to 1.0.
Note that when I tried using 1.0 originally I hit a couple of problems, but
they should be straightforward to overcome.
We have also pu…
-
This is the Epic User Story to create a crawler system for iFiltr, it should be broken down to several other user stories and additional tasks for each user story.
1) Introduction
iFiltr will index …
-
Most nice websites that contain extensive information have their own means to search the site.
Lots of drop in solutions exist:
- htdig
- hyperestraier
- etc., etc.
-
There are currently a few hardcoded references to the name FOIAFeed and the specific nature of the project in the source code. (Off the top of my head, the phrases list, the database name, and the use…
-
**Is your feature request related to a problem? Please describe.**
Most websites need to provide a [`sitemap.xml`](https://www.google.com/sitemaps/protocol.html) for SEO purposes.
**Describe the s…
-
Yes
**Novel URL**: Any on Novelupdates, eg https://www.novelupdates.com/series/yigret
**App Location**: PIP
**App Version**: v3.4.0
## Describe this issue
This source was …
-
Python 3.6, Scrapy 1.5, Twisted 17.9.0
I'm running multiple spiders in the same process per:
https://doc.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process
…
-
Feature:
Add an easy interface (like cPanel's DNS editor) to manage robots.txt "file" output. Only to be working when no static file is present. i.e. Only by the use of filters.
I do not want to o…
-
I have a few specific questions regarding the usage and features of StormCrawler that I hope you could clarify:
Storage of Textual Information: When using StormCrawler with Elasticsearch as outline…