search

ukwa / ukwa-heritrix

The UKWA Heritrix3 custom modules and Docker builder.
10 stars 7 forks source link

Add module to look for well-known URIs when we hit a new host #67

Closed anjackson closed 3 years ago

anjackson commented 3 years ago

For 'survey' crawls, e.g. the domain crawl, it would be great to sweep for some interesting standard URIs:

Others? See https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml

anjackson commented 3 years ago

Those'll do for now.