search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.31k stars 129 forks source link

Change wpull args during a crawl #168

Open Coloradohusky opened 4 years ago

Coloradohusky commented 4 years ago

As the title says, is there any way to change/add wpull args during a crawl?

ivan commented 4 years ago

No, unless you 1) modify grab-site's wpull hook to do something extra, or 2) really want to connect to the manhole and hack away at some internal wpull state

What did you want to change?

Coloradohusky commented 4 years ago

I want to add --retry-connrefused, as I don't want to miss any pages that I get a connection closed on, and I don't want to start over, cause I've been running my crawl for about a week or so now lol