JeremyGrosser / tablesnap

Uses inotify to monitor Cassandra SSTables and upload them to S3
BSD 2-Clause "Simplified" License
181 stars 86 forks source link

Tablechop connection reset #37

Open ethan-eb opened 11 years ago

ethan-eb commented 11 years ago

I'm seeing occasional failures when tablechop attempts to load -listdir.json contents from S3. Since tablechop is meant to run nightly, this error should just be ignored, the next run should pick up the files that didn't get cleaned up.

I can fix this since I'm the only one using tablechop afaik, just needed to log the issue for the next time I'm in there:

Traceback (most recent call last):
  File "/data/syseng/cassandra/eb_tablechop.py", line 67, in <module>
    tablechop.clean_backups(duck_args, log)
  File "/usr/bin/tablechop", line 72, in clean_backups
    jdict = json.loads(ky.get_contents_as_string())
  File "/usr/local/lib/python2.6/dist-packages/boto/s3/key.py", line 1427, in get_contents_as_string
    response_headers=response_headers)
  File "/usr/local/lib/python2.6/dist-packages/boto/s3/key.py", line 1319, in get_contents_to_file
    response_headers=response_headers)
  File "/usr/local/lib/python2.6/dist-packages/boto/s3/key.py", line 1215, in get_file
    for bytes in self:
  File "/usr/local/lib/python2.6/dist-packages/boto/s3/key.py", line 248, in next
    data = self.resp.read(self.BufferSize)
  File "/usr/local/lib/python2.6/dist-packages/boto/connection.py", line 397, in read
    return httplib.HTTPResponse.read(self, amt)
  File "/usr/lib/python2.6/httplib.py", line 538, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.6/socket.py", line 353, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.6/ssl.py", line 96, in <lambda>
    self.recv = lambda buflen=1024, flags=0: SSLSocket.recv(self, buflen, flags)
  File "/usr/lib/python2.6/ssl.py", line 217, in recv
    return self.read(buflen)
  File "/usr/lib/python2.6/ssl.py", line 136, in read
    return self._sslobj.read(len)
socket.error: [Errno 104] Connection reset by peer