Allow users to create Web ARChive (WARC) files

[cgorringe]

There was a suggestion that the extension include a function to be able to save the currently viewed website as a Web ARChive (WARC) file locally on the user's computer. This could be a feature for a future version of the extension.

Relevent Links:

• https://en.wikipedia.org/wiki/Web_ARChive
• https://iipc.github.io/warc-specifications/
• https://github.com/iipc/warc-specifications
• http://bibnum.bnf.fr/WARC/
• https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml
• https://netpreserve.org/resources/WARC_Guidelines_v1.pdf

[Roushangopal]

I am willing to work on this problem, can i? if it's still open

[cgorringe]

Yes, it's still open and no one is working on it currently!

We use Slack internally to communicate. You're welcome to join by emailing our director mark@archive.org who can get you an invite. Just let him know that you'd like to work on this (include a link to this issue).

Thank you.

[cgorringe]

To add to this feature idea, quoting this Tweet by @mekarpeles:

Very few browsers have any reasonable mode for auto-caching websites you visit for offline.

Look, if I visit a web page once, I want it literally forever. And I want all its links archived too (because spacial & temporal locality: I'm likely to revisit a thing & things nearby)

I think this would be a neat feature to be able to auto-save websites that you visit to the local filesystem, which can be viewed later offline. Could be implemented using WARC files?

[sarthakkundra]

@cgorringe

I think this would be a neat feature to be able to auto-save websites that you visit to the local filesystem, which can be viewed later offline. Could be implemented using WARC files?

Maybe a session storage list of all the visited websites? Although those will be links or do you want the whole web page to be stored?

[cgorringe]

Yeah I was thinking actual whole web pages stored locally. A list of visited websites can already be accomplished by not clearing the browser's history.

[sarthakkundra]

@cgorringe session storage will only allow 5 MBs won't that be less for the required task?

[cgorringe]

@sarthakkundra Yeah 5 MBs is way too small for this.

This "File System Access API" appears to be new. I couldn't tell if there were file size restrictions. Haven't tried it and don't know what / if any browser supports it yet. https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API https://web.dev/file-system-access/

[sarthakkundra]

@cgorringe this looks interesting. I'll have a look. Meanwhile is there a Slack, Gitter, IRC channel etc that the community uses for communication or everything happens on Github only? I'd love to join if there's one to discuss other ideas as well

[public-rant]

Has there been any progress on this issue? Anything interesting to report?

[cgorringe]

@public-rant No progress currently, sorry.