search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.35k stars 134 forks source link

Does gs-server really need to run? #123

Closed hofmand closed 6 years ago

hofmand commented 6 years ago

It keeps shutting down on its own, but other than the Could not connect to ws:// messages, grab-site seems to work fine. So is gs-server really needed?

ivan commented 6 years ago

You don't really need gs-server running, but I think it is very helpful for monitoring crawls and creating ignore regexps.

Do you know why gs-server is shutting down on its own? Do you see any crash messages from the program? Which operating system? Asking because I have never seen it shut down. Thanks.

ivan commented 6 years ago

@hofmand ping. any information on the gs-server crashes?

hofmand commented 6 years ago

Maybe it's related to the way I wasn't detaching properly from the terminal, or the fact that I was connecting to the server from my cell phone. Now I'm using screen and I haven't seen the issue recently.

ivan commented 6 years ago

Yeah, if you get rid of the terminal, I think Python will eventually crash or block on not being able to write to stdout. Glad to hear it works with screen.