search

hartator / wayback-machine-downloader

Download an entire website from the Wayback Machine.
Other
5.24k stars 691 forks source link

Some downloaded files are gzip stream #259

Open FoxAhead opened 11 months ago

FoxAhead commented 11 months ago

I used this great tool to download the site http://web.archive.org/web/20230713110210/http://users.tpg.com.au/jpwbeest/. At first glance everything went well, but then I found out that some downloaded files, regardless of extension, were saved as GZIP stream. Some were fine. The result was consistently repeated on repeated downloads. It was about 30 "corrupted" files out of total 245.

Examples of gzipped files (The first two bytes 1F 8B are gzip magic number, and the third 08 is deflate compression)

image image

I would like to know what causes this to happen. Is it a bug or peculiarities of this site or the whole Wayback Machine? Is it possible to fix it?

So far I've solved this problem with a simple python script that scans the files in the directory, and if the file has signs of a gzip stream, decompresses it, or otherwise just copies it to the output folder. Thanks!

lihaohong6 commented 11 months ago

Waybackmachine might have changed how their api works. I'm downloading webpages archived in the past two months, and all of them end up being gzip files. I suspect that the change happened sometime during the last two months.

grandpa1946 commented 11 months ago

i am also having this issue

Na-x4 commented 10 months ago

i think reverting #34 will solve this problem.

tudoujunha commented 4 months ago

i think reverting #34 will solve this problem.

You are right. This will solve the problem: https://github.com/hartator/wayback-machine-downloader/issues/267#issuecomment-1868090089 I recommend this: https://github.com/hartator/wayback-machine-downloader/pull/280#issue-2124403834