support Google App Engine's URL fetch service

[GoogleCodeExporter]

What steps will reproduce the problem?
1. use Google's App Engine SDK
2. install couchdb-python into your new project
3. import couchdb
2. attempt to connect to an external couchdb server with couch = 
couchdb.Server('https://couchdb.example.com/')
3. exception is raised

What is the expected output? What do you see instead?

Google has their own url fetch service in their app engine. 
<http://code.google.com/appengine/docs/python/urlfetch/overview.html>. Other 
than that, they appear to prevent applications from making external socket 
connections. It appears to me that since this code uses couchdb/http.py, it 
won't work on Google's app engine. Here's the stacktrace:

Traceback (most recent call last):
  File "/home/amoore/dev/google_appengine/google/appengine/ext/webapp/__init__.py", line 511, in __call__
    handler.get(*groups)
  File "/home/amoore/dev/emradmin/main.py", line 13, in get
    userdb = couch['user']
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 137, in __getitem__
    db.resource.head() # actually make a request to the database
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 377, in head
    return self._request('HEAD', path, headers=headers, **params)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 419, in _request
    credentials=self.credentials)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 239, in request
    resp = _try_request_with_retries(iter(self.retry_delays))
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 196, in _try_request_with_retries
    return _try_request()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 211, in _try_request
    if conn.sock is None:
AttributeError: HTTPConnection instance has no attribute 'sock'

If urllib, urllib2 or httplib were used instead, I think this would work.

What version of the product are you using? On what operating system?
python 2.5.2, Google App Engine, linux

Original issue reported on code.google.com by acmo...@gmail.com on 30 Aug 2010 at 8:51

[GoogleCodeExporter]

couchdb.http is actually just a layer over httplib so, in theory, it should 
work. Unfortunately, couchdb.http uses an undocumented 'sock' attribute of 
httplib.HTTPConnection that the Google API doesn't seem to provide.

'sock' is primarily used to re-open dead, cached connections but I don't see 
any other obvious, i.e. documented, way to test if a connection is open. I 
guess we could just rely on the retry code picking it up.

Note: couchdb.http also uses conn.sock.sendall but I suspect that could easily 
be replaced with conn.send.

Original comment by matt.goo...@gmail.com on 30 Aug 2010 at 9:52

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Aha. That makes sense. If you come up with a potential fix, and you don't want 
to jump through the hoops of getting an App Engine account, I can try it out 
for you. In the meantime, I'll try to hack my way into a "fix".

Thanks!

Original comment by acmo...@gmail.com on 30 Aug 2010 at 9:56

[GoogleCodeExporter]

Thanks for the pointers. I've attached a patch that removes the calls to 
'sock'. The test suite seems to do about the same (there are 3 failures before 
and after the patch), and this patch works in Google's App Engine. I don't 
really know how to test the effects on timeouts.

Original comment by acmo...@gmail.com on 31 Aug 2010 at 4:02

Attachments:

• couchdb_google.diff

[GoogleCodeExporter]

hrm. It looks like they don't implement a 'isclosed' method, either:

  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 292, in __iter__
    return iter([item.id for item in self.view('_all_docs')])
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 984, in __iter__
    return iter(self.rows)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 1003, in rows
    self._fetch()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 990, in _fetch
    data = self.view._exec(self.options)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 880, in _exec
    _, _, data = self.resource.get_json(**self._encode_options(options))
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 394, in get_json
    data = json.decode(data.read())
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 96, in read
    self.close()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 100, in close
    while not self.resp.isclosed():
  File "/home/amoore/dev/google_appengine/google/appengine/dist/httplib.py", line 231, in __getattr__
    return getattr(self.fp, attr)
AttributeError: StringIO instance has no attribute 'isclosed'

This means you can't iterate over a database.

I'll see if I can work around this, too.

Original comment by acmo...@gmail.com on 31 Aug 2010 at 10:00

[GoogleCodeExporter]

So, what about GAE support?

Original comment by polestn...@gmail.com on 13 Sep 2010 at 3:35

[GoogleCodeExporter]

acmoore: that patch looks solid. Can you give me your name and an email address 
so we can credit you on the changeset?

Original comment by djc.ochtman on 19 Sep 2010 at 1:15

[GoogleCodeExporter]

You can credit it to: Andrew Moore <amoore@mooresystems.com>. Thank you!
I'm also willing to assign copyright to one of you, in case that helps with 
future maintenance.

I have a patch for the 'isclosed' problem, too. It's necessary in order to 
iterate over databases inside the GAE. I'll find it, clean it up, and submit 
it, too.

Original comment by amo...@mooresystems.com on 19 Sep 2010 at 2:08

[GoogleCodeExporter]

Pushed as r25df81c5d704. Would be great if you come up with the other patch! 
Can you also show us what kind of test suite failures we get on GAE? Also, 
apparently we don't have iterating over databases covered in the test suite? 
Would be nice to fix that, too.

Original comment by djc.ochtman on 19 Sep 2010 at 3:01

[GoogleCodeExporter]

Thanks for pushing out that previous patch.

Here's the patch I'm using to get around the problem with .isclosed(). Now that 
I look at it, I'm not sure that it is only working because I always happen to 
get all of the data in the first read or not. Works for me, though.

Original comment by amo...@mooresystems.com on 19 Sep 2010 at 4:12

Attachments:

• isclosed.diff

[GoogleCodeExporter]

This issue has been migrated to GitHub. Please continue discussion here:

https://github.com/djc/couchdb-python/issues/147

Original comment by djc.ochtman on 15 Jul 2014 at 7:19