search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.35k stars 134 forks source link

Enhancement idea: show failed jobs on the dashboard until gs-server is restarted #84

Open ethus3h opened 8 years ago

ethus3h commented 8 years ago

As in the title. This would enable easier noticing when something goes wrong like in #52....

ivan commented 8 years ago

This wouldn't help you notice a Segmentation fault because grab-site can't send a message to gs-server when it segfaults. (ArchiveBot has a different design where a parent process is responsible for sending messages, so it can capture the stdout and stderr regardless.)

ethus3h commented 8 years ago

Ah, hmm, oh well. Maybe the gs-server could remember the most recent line of each job, so that stuck ones would become evident?

ethus3h commented 8 years ago

Strike my previous idea; I think it would slow down gs-server probably.... How about, the gs-server could remember the address and identifier of each job?

ivan commented 8 years ago

Remembering the last few events for each job wouldn't slow down gs-server. It's probably not too hard to implement either (just use a Python deque).