Non-HTML files given .html file extension with HTMLFirst enabled

AnyOldName3 commented 10 months ago

I'm archiving a website and won't need it to work offline, so got rid of the +*.jpg etc. rules that were causing externally-hosted images to be included. This changed a few images referenced by a relatively small number of pages to change their file extension from .ico, .jpg or .png to .html. I'm concerned about this as the archive will eventually be hosted on a webserver that uses file extensions to automatically determine MIME types, and I anticipate problems if there's a mismatch.

The file extensions and MIME type provided by the original webserver are correct, so I don't know why it's decided to make up new ones.

AnyOldName3 commented 10 months ago

Some relevant lines from logs etc. for one of the affected files (vsg-small.png):

`hts-log.txt`

22:50:25    Warning:    Unexpected 412/416 error (Requested Range Not Satisfiable) for https://www.openscenegraph.com/images/vsg-small.png, 'C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.html' could not be found on disk

`new.lst`

[www.openscenegraph.com/images/vsg-small.png]
[www.openscenegraph.com/images/vsg-small.html]

`new.txt`

17:38:50    314/314 ------  416 error ('Requested%20Range%20Not%20Satisfiable') text/html   date:Wed,%2022%20Nov%202023%2016:57:05%20GMT    https://www.openscenegraph.com/images/vsg-small.png C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.png   (from https://www.openscenegraph.com/)
23:00:37    6895/6895   ---M--  200 added ('OK')    image/png   etag:%221aef-57b663281bf20%22   https://www.openscenegraph.com/images/vsg-small.png C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.html  (from https://www.openscenegraph.com/images/vsg-small.png)

You can view everything from the time it didn't do this (and I included a bunch of stuff from domains I didn't need) at https://github.com/AnyOldName3/OpenSceneGraphDotComBackup/tree/main/OpenSceneGraph/hts-cache

AnyOldName3 commented 10 months ago

I've done some digging, and haven't determined whether the 416 was caused by HTTrack submitting a dodgy request or by the server doing something wrong. As the file has the right contents and the log mentions the right MIME type, I think it's plausible that the response body was still the correct file and response header still had the right MIME type, and HTTrack automatically changed the file extension because the status code represented an error.

AnyOldName3 commented 10 months ago

Actually, that's not right - the line that mentions the 416 error mentions text/html and it's only the later one that has a 200 status that mentions image/png. That would suggest to me that the server's setting the wrong MIME type when it generates an error. I still don't know whether the error is the server's fault or HTTrack's.

AnyOldName3 commented 10 months ago

Definitely down to an intermittent fault as it's a different set of files affected when rerunning the mirroring process again. I've still not managed to access the server logs, so have no more information about whether it's down to malformed requests or incorrect handling of well-formed requests.

AnyOldName3 commented 10 months ago

I've looked a bit more, and it's apparently also affecting loads of gzips that didn't have the problem the first time I attempted the mirroring process. As far as I can tell (I committed to a Git repo after the first attempt, so it should be accurate), the only things that changed on HTTrack's end are the following winprofile.ini changes:

HTMLFirst=1 (was HTMLFirst=0, which the GUI says is slower)
WildCardFilters=+*.css +*.js -ad.doubleclick.net/* -mime:application/foobar (was WildCardFilters=+*.png +*.gif +*.jpg +*.jpeg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar, which caused images from other domains to be included)
CurrentUrl=https://www.openscenegraph.com/%0d%0alists.openscenegraph.org%0d%0alists.openscenegraph.org/pipermail/osg-users-openscenegraph.org%0d%0awww.openscenegraph.com%0d%0awww.openscenegraph.org (was CurrentUrl=https://www.openscenegraph.com/, which required me to manually include more URLs in interactive mode).

I should probably clarify that the old.txt and old.zip mentioned below are for a previous partially successful run of the process with the newer settings, not the run with the old settings that I linked to a few posts ago.

Some of these tie in with lines in new.txt like

20:20:30    76352/76352 ---M--  200 added ('OK')    application/gzip    etag:%2212a40-574b8b6a9f35d%22  http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz   C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.html  (from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)

In the old.txt, the same URL is mentioned in these two lines:

17:40:31    76352/76352 ---M--  200 added ('OK')    application/gzip    etag:%2212a40-574b8b6a9f35d%22  http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz   C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gzip  (from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)
22:50:33    314/314 UR-MC-  416 error ('Requested%20Range%20Not%20Satisfiable') text/html   date:Wed,%2022%20Nov%202023%2022:50:33%20GMT    http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz   C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gzip  (from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)

Looking in new.zip, there's an empty file with the same name, but in old.zip, it's there twice (yay, weird quirks of the zip underspecified zip format), once as an empty file, and once containing:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>416 Requested Range Not Satisfiable</title>
</head><body>
<h1>Requested Range Not Satisfiable</h1>
<p>None of the range-specifier values in the Range
request-header field overlap the current extent
of the selected resource.</p>
</body></html>

I guess that debunks the theory that the 416 errors still led to the same file contents being served. It also makes it look like the presence of these files in the cache zip from the previous run poisons the next run, even if the file becomes available again.

AnyOldName3 commented 10 months ago

I've just noticed that I've been using Download web site(s) + questions instead of * Update existing download, which I imagine might not have been the best idea for the runs where I didn't delete everything first to start with a clean slate.

AnyOldName3 commented 10 months ago

Doing a fresh run with the same settings generated no 416 errors at first, then towards the end of the process, the first number in the Links scanned: 12345/12345 (+1234) bit reached the same value as the second number, and started counting again from zero. During this phase, a significant percentage of the files fetched generated 416 errors.

As it was a clean run, this can't have been caused by the cache from a previous run poisoning the next run. I don't think this was the setting to deal with HTML files first as there are plenty of non-HTML files before this point in new.txt.

AnyOldName3 commented 10 months ago

Today I tried running this again with the option to fetch HTML files first disabled, like it had been for my initial, successful run. I hit no HTTP 416 errors, and all files were given the correct extension. I still don't know whether the 416 errors were caused by malformed requests from HTTrack or by the server misbehaving, but at least this is no longer giving me grief. I also found it ran in about half the time with the option disabled, which isn't what the tooltip or documentation suggested.

xroche / httrack