system_limit using change() function

[GoogleCodeExporter]

When many change operations are sent in a row on my Mac with Python
2.6.1, CouchDB 1.0.1, and CouchDB-Python 0.8, I get the following
after about 32,600 calls to change():

Traceback (most recent call last):
 File "couchdb_blast2.py", line 28, in <module>
   test1()
 File "couchdb_blast2.py", line 10, in test1
   db.changes()
 File "/Library/Python/2.6/site-packages/CouchDB-0.8-py2.6.egg/
couchdb/client.py", line 801, in changes
 File "/Library/Python/2.6/site-packages/CouchDB-0.8-py2.6.egg/
couchdb/http.py", line 393, in get_json
 File "/Library/Python/2.6/site-packages/CouchDB-0.8-py2.6.egg/
couchdb/http.py", line 374, in get
 File "/Library/Python/2.6/site-packages/CouchDB-0.8-py2.6.egg/
couchdb/http.py", line 419, in _request
 File "/Library/Python/2.6/site-packages/CouchDB-0.8-py2.6.egg/
couchdb/http.py", line 310, in request
couchdb.http.ServerError: (500, (u'unknown_error', u'system_limit'))

Here is the code, where the test database contains a single document:

import couchdb

def test1():
   couch = couchdb.Server('http://localhost:5984')
   db = couch['test']
   i = 0
   while(True):
       db.changes()
       i = i + 1
       if i % 100 == 0:
           print i

I initially thought that I am not giving CouchDB time to recover, but
when I add a time.sleep(0.1) in the loop it still fails after the same
number of iterations. When I change the test to use httplib directly
it seems to work indefinitely:

from httplib import HTTPConnection

def test2():
   conn = HTTPConnection('localhost:5984')
   conn.connect()
   i = 0
   while(True):
       conn.putrequest('GET', '/test/_changes')
       conn.endheaders()
       conn.getresponse().read()
       i = i + 1
       if i % 100 == 0:
           print i

When I use netstat while it's running, in both test1 and test2 I only
see a few entries like this related to couchdb:

tcp4       0      0  localhost.5984         localhost.64944
ESTABLISHED
tcp4       0      0  localhost.64944        localhost.5984
ESTABLISHED

Original issue reported on code.google.com by jeff.bau...@kitware.com on 18 May 2011 at 9:18

[GoogleCodeExporter]

previous discussion:
https://groups.google.com/forum/#!topic/couchdb-python/9ybLFoskHBI

Closing connection is the solution, but a little brutal. After two hours of 
testing I could say that problem is in one thing: using cached connection for 
_changes feed. Error occurred only on conn.getresponse.read() calls, but 
doesn't if response haven't be readed. Chunked list responses doesn't reproduce 
this error. However, I've got it for feed=longpool query parameters using only 
test2 function. Going to ask CouchDB community about correct behavior in this 
case.

Original comment by kxepal on 19 May 2011 at 1:53

[GoogleCodeExporter]

Seems this is not couchdb-python bug, nor CouchDB itself [1] and is normal 
behavior for _changes feed. There is nothing could be done, I suppose.

[1] https://issues.apache.org/jira/browse/COUCHDB-1171

Original comment by kxepal on 23 May 2011 at 9:17

[GoogleCodeExporter]

Thanks for looking into it. I suppose this is a good use case on how *not* to 
do polling with CouchDB, as it is sure to fail after a certain number of 
iterations.

The CouchDB issue comment "feed=longpoll or feed=continuous are the only 
reliable ways to make long-lasting HTTP connections", and the fact that the 
pure HTTP does not fail when using the default feed, makes me wonder if there 
is still something couchdb-python could do differently in the default feed case 
to make it work, since it sounds like the default feed should not make 
long-lasting HTTP connections. But regardless, I plan on changing my code to 
not use loops like this, and this may not be a critical fix for others.

An aside - In the code you posted in the CouchDB issue you used the string 
"longpool" instead of "longpoll". I assume you actually used the correct string 
in your testing.

Original comment by jeff.bau...@kitware.com on 24 May 2011 at 2:24

[GoogleCodeExporter]

> An aside - In the code you posted in the CouchDB issue you used the string 
"longpool" instead of "longpoll". I assume you actually used the correct string 
in your testing.

Shame on me, that wasn't a typo ): However, I wounder why it had reproduced 
invalid behavior. Any way, for most real cases, calling db.changes produce too 
much output which makes memory run faster, so better to avoid such situations.

Original comment by kxepal on 25 May 2011 at 9:30

[GoogleCodeExporter]

Just to let you know that I'm now reasonably sure this is caused by a bug in 
CouchDB. I'll be updating https://issues.apache.org/jira/browse/COUCHDB-1171 
issue shortly.

Original comment by matt.goo...@gmail.com on 26 May 2011 at 10:47

[GoogleCodeExporter]

Thanks Matt! I've checked everything except why exception have raising and 
CouchDB logs for test2, because expected that behavior should be the same as 
for, actually, invalid feed parameter. I'll try to be more careful next time(:

Original comment by kxepal on 27 May 2011 at 5:41

[GoogleCodeExporter]

My fix has been committed to CouchDB and should will be in the 1.1.0 release. I 
don't think we need to change anything in couchdb-python.

Original comment by matt.goo...@gmail.com on 28 May 2011 at 7:55

• Changed state: WontFix