search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.34k stars 134 forks source link

Backslash to Forward slash correction #199

Open acrois opened 3 years ago

acrois commented 3 years ago

It seems that browsing to a URL like this: https:// possibility.com/Clearing/Images\brnbagbk.gif (github encodes these URLs so I had to put a space after "https://") with a backward slash will resolve properly in browsers (latest Google Chrome and Firefox tested) by changing it to a forward slash when the URL is processed. It will automatically replace the backward slashes with forward slashes.

When it comes to storing the response in WARC as well as tools retrieving the correct URL (similar to how a browser would correct the request) I am not sure of the implications. I can imagine a process where a URL that resolves 404 with a backward slash in it can be retried with the backslashes replaced with forward slashes.

From the dashboard, when viewed in Google Chrome: image image

What is the best way to handle this in grab-site?

TheTechRobo commented 2 years ago

I might send a PR, but I don't really understand the codebase that well. 🤕

ivan commented 2 years ago

Browsers convert backslashes to slashes when they parse hrefs. They also do other odd things there, like trim leading and trailing whitespace.

I believe the right place to implement this would be in the href parsing in ludios_wpull.