Suspicious logger exception

[GoogleCodeExporter]

While trying to run memcachedb, this occurs:

Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/handlers.py", line 72, in emit
    self.doRollover()
  File "/usr/lib/python2.6/logging/handlers.py", line 129, in doRollover
    os.rename(self.baseFilename, dfn)
OSError: [Errno 13] Permission denied
appscale-image1 - - [09/Dec/2009 04:06:43] "POST dbtest:1 HTTP/1.1" 200 -
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/handlers.py", line 71, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python2.6/logging/handlers.py", line 145, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/lib/python2.6/logging/handlers.py", line 71, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python2.6/logging/handlers.py", line 145, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file
(and so on)

The line in the middle of the exception, 

appscale-image1 - - [09/Dec/2009 04:06:43] "POST dbtest:1 HTTP/1.1" 200 -

, shows that the database can still be written to, so the database and
related components have not failed. Need to investigate what causes this
problem, its effects on the system, and how to resolve it.

Original issue reported on code.google.com by shattere...@gmail.com on 9 Dec 2009 at 6:48

• Blocked on: #12

[GoogleCodeExporter]

Believed to be this bug:

http://bugs.python.org/issue1425127

Will continue to investigate.

Original comment by shattere...@gmail.com on 10 Dec 2009 at 2:01

• Changed state: Started

[GoogleCodeExporter]

Turns out to be not the bug above. This exception occurs because memcachedb 
logs all
database accesses, which under high load quickly fills up the file system 
(causing
the logger to fail and write the exceptions above).

There appears to be no 'quiet' option for memcachedb, will continue to 
investigate.

Original comment by shattere...@gmail.com on 10 Dec 2009 at 8:41

[GoogleCodeExporter]

MemcacheDB offers a flag to cleanup old log files, which solves this problem 
for now.
It won't be fully solved until the following is done:

1) Persistence (issue X), to get bigger disks in EC2/Euca via EBS.

2) Added logging to catch when the file system is full and report it to the user
appropriately (new tool or put it in describe-instances?)

Original comment by shattere...@gmail.com on 7 Jan 2010 at 7:47

• Changed state: Accepted

[GoogleCodeExporter]

Closed for now; not considered to be a true issue since it's a side effect of 
the HD
filling up.

Original comment by shattere...@gmail.com on 27 Jan 2010 at 5:34

• Changed state: WontFix