Crawl eventually becomes nothing but "Disconnected from ws:// server:"...

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Other

1.31k stars 129 forks source link

Crawl eventually becomes nothing but "Disconnected from ws:// server:"... #152

Closed BradCoffield closed 5 years ago

BradCoffield commented 5 years ago

Hi there,

I'm getting this command line output sometimes through the crawl and last night after letting a full crawl of our site go overnight I was getting just it. Though once I closed the process and inspected the warc it seemed to be in good shape...

Disconnected from ws:// server: ConnectionRefusedError(111, "Connect call failed ('127.0.0.1', 29000)")

I have no idea what this means. Should I be adding that port to an ignore list? Thanks!

ivan commented 5 years ago

That's grab-site trying to connect to gs-server. It's harmless if grab-site fails to connect, though it is unexpected if gs-server is running.

BradCoffield commented 5 years ago

Interesting. Okay, thank you!