search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.34k stars 134 forks source link

Resuming a WARC after hard "No space left on device" error message? #210

Closed Preservation-Quest closed 2 years ago

Preservation-Quest commented 2 years ago

Ran out of space and although I understand how to download to a non-default directory, there isn't any information on resuming a mirror / WARC file.

ivan commented 2 years ago

It's not really supported yet - see #58

I recommend adapting and running https://github.com/ArchiveTeam/grab-site/blob/14c3bbdf7156a923c3b5baeef60b4e9fa3ef363c/extra_docs/pause_resume_grab_sites.sh in the background to pause grab-site processes when low on disk.