sedimentation-fault
commented
7 years ago Another example: what happens if you had to interrupt getpapers with CTRL-C?
I have an intermittent internet connection - the router may simply reconnect (as a result of some other operation, accident, or just due to the provider's daily change of a dynamic IP address) at no notice to a client like getpapers. And what does getpapers do in such a case? It hangs indefinitely - NOT for a timeout! I have seen it hanging for hours! In such cases, only CTRL-C helps.
What is the status of the files downloaded so far then? Especially, how do you check the integrity of the file that getpapers was downloading when it received the CTRL-C interrupt? Do you compare sizes somehow? I doubt it.
You need to resume gracefully form the point you stopped. But for this to work, the web server has to support "byte ranges". If it does not, it will NOT respond with
206 Partial Content
-- in which case getpapers should delete the partially downloaded file and retry from scratch. And what do you do if the file was wholly downloaded in a previous run? Do you try to re-download it completely or partially? In the later case, you will get a
416 Range Not Satisfiable
error - which practically means "you have either this file completely, or some other file with this name, which is larger than this (or at least as large)".
I would like to know that all this is handled by getpapers gracefully - and have peace of mind while downloading.
An HTTP server may throw a lot of HTTP errors at a user agent's face - you can see a list of them at List of HTTP status codes (Wikipedia). IMHO a program like getpapers should be able to handle ALL of them - but I resist the temptation to file a bug for each one of them that has been left unhandled. Instead, I view this thread as the collective place to discuss them all.
It's not only "connection reset by peer" (see https://github.com/ContentMine/getpapers/issues/155) errors that I am concerned about, though. It's the whole gamut of HTTP errors that I wish to see handled in a graceful way.
What about, for example, the error:
500 Internal Server Erroror
429 Too Many Requestswhich I do encounter every now and then? In case of the latter: is there a "backoff strategy" put in place, i.e. does getpapers retry after a variable time intervall (e.g. after 2, 4, 8, 16, 32 seconds...)? That is, 2 seconds after the first occurrence, 4 seconds after the second one in-a-row, 8 seconds after the third consecutive one and so on?
You can't hammer on a web server and expect it to "just work" - getpapers has to be as flexible as a human at handling HTTP errors. Pure retrying is not enough. You need an exponential backoff strategy for every error that is not permanent (most 5xx errors and some 4xx ones too, like the 429 error above).
And - believe me - ALL errors in the list do happen if you download often enough - as one does with getpapers.