search

webrecorder / replayweb.page

Serverless replay of web archives directly in the browser
https://replayweb.page
GNU Affero General Public License v3.0
707 stars 58 forks source link

Unable to load a big WARC (18GB) #93

Open Scorpin opened 2 years ago

Scorpin commented 2 years ago

Hi, I'm trying to read a very big WARC, of 18GB as I said in the title, and using the desktop version for Windows, the load stops (in fact, the app only change to a white screen, with the menus and the windows still active, so I supose it just make crash the algorithm of loading files) at the 21%, 4.2GBs in the progress bar, and 7.4GBs of effective RAM used for the app in that moment. I supose the app it's trying to load all the file in the RAM, so... It's something a bit troublesome when you are trying to use big files like this one (I have 16GBs of RAM btw, and well, when the load stops the total RAM was still at 80% aproximatelly, so isn't a problem of "no more RAM space", in this case at least). In older versions of this app, like the Web Recorder Player (x64), the WARC can be loaded without problems, except the app enter in an infinite "Indexing..." state and finally crash in a similar way when I try to access to the search field, but all the links are loaded in screen and I can access them if I just go to search them manually, so... I supose the problem isn't only the size of the file, but the way you manage to load the files. There is some way to fix this from my own? Or can you try to fix it in a next release?

Thank you.

urdaddyVlex commented 2 years ago

same problem here i wonder if you have solved it？

Scorpin commented 2 years ago

same problem here i wonder if you have solved it？

Oh sorry, I didn't reply you... I just used the older version of Web Recorder Player without using their search option, just going manually to the entry I need, and even a bit slow with a so big file, it works for me... I hope it's enough for you.

Greetings.

urdaddyVlex commented 2 years ago

i tried 1.1.2 and it worked!! search tools didn't work well tho. anyways thanks for your help

talyacooper commented 2 years ago

I have this problem as well. I've consistently had issues opening large WARC files on ReplayWeb.page. Downloading the files and opening them in the older desktop version of Web Recorder Player worked for me, but since I updated to MacOS Monterey I'm no longer able to use the WebRecorder Player. Wondering if anyone has any suggestions?

tmob03 commented 5 months ago

same problem here i wonder if you have solved it？

Oh sorry, I didn't reply you... I just used the older version of Web Recorder Player without using their search option, just going manually to the entry I need, and even a bit slow with a so big file, it works for me... I hope it's enough for you.

Greetings.

What version are you using? It won't even load 6gig warcs for me.