Archive sites in addition to submitting URIs

[machawk1]

One of the use cases in https://github.com/webrecorder/warcit is to grab a site's contents using wget then running the tool to create a WARC file from the local file contents. It would be useful for a tool called, "archivenow" to do more than submit URIs, rather, to perform some form of archiving itself.

I would like to propose replicating this model from the archivenow tool but in a single command. For example, running archivenow --warc=news.warc --agent=wget --ia http://cnn.com would use wget to create a WARC of cnn.com and store it locally at news.arc but also submit the URI to IA.

[maturban]

It is really nice to have "archivenow" create WARCs locally, not just pushing URLs to other archives. It is like pushing URLs into local archive in addition to the remote ones. I will definitely implement this as soon I can. Because this is written in Python, I would suggest using the module "requests" or any other Python module instead of "wget"! what do you think?

[machawk1]

You will need to chase down all of the embedded resources w/ requests. wget does this for you and has native support for WARC output. If there was a Python equivalent of @N0taN3rd's https://github.com/n0tan3rd/node-warc, that would work well, too.

[N0taN3rd]

For Python side of controlling chrome without handling the raw websockets

• Python 3.6 asyncio library chromewhip. I like this one
• PyChromeDevTools
• pychrome

[maturban]

I am closing this as we already included creating WARCs by Wget and Squidwarc