[Suggestion] Give the option to crawl an entire website + other features

[witedragen]

I'd like to capture an entire website using webrecorder desktop instead of having to manually click each individual page. It would be nice to:

• have the option to split the WARC in x GB pieces if it were to grow that big
• import a regex set of ignored urls (to prevent something like &rs=5&rs=5&rs=5&rs=5)
• choose how fast or slow the pages are downloaded (to prevent being banned on some sites)
• set the number of retries on webpage/resource errors
• choose what user agent to use from a list or give the user the option to manually specify one

maybe in an "advanced" dropdown menu for advanced users

Good programme so far !!

[kbhasi]

• choose what user agent to use from a list or give the user the option to manually specify one

Yes! Electron is based on Chromium, which is what Google Chrome is based on, so it loads pages the same way Chrome does, however, I prefer to use Pale Moon as my default Web browser, followed by Firefox, and then Edge (not the newer Chromium based release) if I'm on Windows, as I'm that much of a rebel. As such, I want to use a Pale Moon or Firefox user agent, rather than a Google Chrome user agent.

My current workaround is to open the Chrome developer tools and set the user agent in the "network conditions" tab, but that only works once I load a website, so I have to "preview" first.

Edit: Additionally, I'd want the option to set the "Do Not Track" flag too.

[makew0rld]

@ikreymer Any idea whether we'll see this feature? It would be awesome.

And if I'm thinking correctly, the only way to recursively archive a website right now would be to collect the urls and archive them all at once with pywb, right?

Edit: Looks like maybe I should use browsertrix for that?

[kbhasi]

I found even more issues with trying to change user agents in the current release. When switching from preview to record, the developer tools window closes as it seems the app is closing and opening a new Chrome tab.

A bigger reason why I want to change user agents is because I want to "record" the old YouTube web interface before Google deletes it later in March 2020 and keeps the current interface that has larger elements (thank goodness for NoSquint!), and a similar thing for Twitter's old "Twitter Web Client" interface, as opposed to the current "Twitter Web App" interface.

[kbhasi]

A bigger reason why I want to change user agents is because I want to "record" […] Twitter's old "Twitter Web Client" interface, as opposed to the current "Twitter Web App" interface.

Well, I can't believe I wasn't able to record that, especially while logged in, before Twitter killed it off. I need to get myself to look away from this and onto other things. I tried to launch Webrecorder Desktop, but it seems to launch 4 processes, and stays dormant without showing itself.