Closed benoit74 closed 4 months ago
Indeed ; that used to be handled by the SW via the Headers I believe. Interesting discussion ahead!
- redirect 301, 302, 307 and 308 should lead to the creation of an alias in most situations (only redirections leading to the same ZIM path should be ignored, e.g. http to https redirection since we drop the scheme)
Not an alias, a redirect. ZIM redirects can only be internal, so redirects to external resources should be just ignored. Clearly an important thing to fix.
I propose that we first implement a first enhancement which will cover 80% of the situation.
WDYT?
add ZIM redirects for 301, 302, 307 and 308 to an internal URL
Of course, this is a regression
all other 2xx and 3xx return codes, and 301, 302, 307 and 308 to an external URL are either ignored (with a warning in the log) or stop the scraper
203
, 204
and 205
can be easily supported. Other 20x
are too niche or depends on headers (206
especially).
Regarding redirects to external URI, we should not include them in the ZIM. This is the reasonable behavior: if the user is online when reading, it could work. If he's not, it will fail in the most expect-able way possible. I see no reason to stop the crawler for this. Warning has limited value but it's cheap so let's do it.
all 1xx, 4xx and 5xx are simply ignored with a warning in the log (not sure we have any 1xx or 5xx in the WARC records in fact)
There should be 5xx
records in WARCs but not for the seed URL now that we use the option to fail in that case. In zimit1 we chose to not include them in the ZIM as those are considered not actionnable by end-user and should thus be considered external ; offering possibility of online access.
Do you mean that for 203
, we can simply store the content? Same is probably true for 202
then.
I don't get what you want to do with 204
(simply ignore it? no content is ... no content ... do you intend to add an empty ZIM entry just to indicate that URL is valid?) and 205
(simply store content and forget about the request from upstream to reset requesting document?)
Yes, that's exactly what would be in the WARCs and there's no reason to discard them as those don't rely on headers. We're probably talking about incredibly small number or entries anyway
We're probably talking about incredibly small number or entries anyway
Definitely, but better to fix as many cases as possible at once when they are simple rather than come back many time to it, you are right about mentioning these cases should be handled as well.
203
,204
and205
can be easily supported. Other20x
are too niche or depends on headers (206
especially).
So, what happens with 206 partial content responses if they have been recorded in the WARC Headers and Responses? In the case of zimit1, we (in KJS) have code to set our Response headers appropriately if we receive a WARC Response Header indicating a range.
For zimit2, I still need to know what to expect in the Response body, as we have to reconstruct Response headers in our Service Worker in KJS in order to reply to requests made by the browser. Are partial responses just discarded, or are they converted to full content responses, for example?
For now, all WARC records with an "unmanaged" HTTP response code will be ignored by warc2zim2 (e.g. 206
).
This means that if there is not other WARC record with another HTTP response code in the WARC archive, the corresponding URLs found in documents will be considered as external to the ZIM and hence pointing to the online resource.
If the user is online, the website will be able to fetch the payload. If user is offline, result will depend on how the website handles a failing call to this URL.
If the user is online, the website will be able to fetch the payload.
That would be risky and probably blocked by CORS, definitely blocked by the sandbox at least in KJS, but I'm pretty sure also in Kiwix Desktop.
I think how we currently handle 206 requests that the user makes (e.g. if the user moves the slider in a video quickly to another part of the video) is that we recognize the browser has made a 206 request, and we reconstruct the response as a 206 starting at the requested starting point, and ending at the end of the requested medium (video). This minimizes requests if the user skips around.
Maybe this is sufficient also to deal with any dynamic partial content requests made by the JS player (if it does stream that way). So, let's not worry about it unless we hit such issues in the wild. 🙂
There is two (at least) problem to consider:
known_url
set, while many won't be there because there is no content (we create a ZIM record only if WARC record size is > 0) ; probably only few 2xx and 3xx will actually make it to the ZIM