feat(revisit): Added new Robots Redirect check - Githubissues

mirrorweb / pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

https://pypi.python.org/pypi/pywb

GNU General Public License v3.0

1 stars 2 forks source link

feat(revisit): Added new Robots Redirect check #42

Closed hyl closed 2 years ago

hyl commented 2 years ago

Description

Introduces a new raise_on_robots_redirect function to check and resolve any records that incorrectly have a http->https robots.txt default redirect.

Motivation and Context

Screenshots (if appropriate):

Types of changes

[ ] Replay fix (fixes a replay specific issue)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added or updated tests to cover my changes.
[ ] All new and existing tests passed.