http.client broken on google app engine

[snarfed]

first off, thanks for future! it's hugely valuable.

i noticed that the aliases for http.client don't work in google app engine, at least not in dev_appserver.py. (#231 is an older, similar app engine compability issue.) here's the exception:

Traceback (most recent call last):
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/ryan/src/oauth-dropins/foo.py", line 9, in <module>
    import http.client
  File "/Users/ryan/src/oauth-dropins/local/lib/python2.7/site-packages/http/client.py", line 11, in <module>
    from httplib import (HTTP_PORT,
ImportError: cannot import name _CS_IDLE

to reproduce, create these two files: app.yaml:

api_version: 1
runtime: python27

- url: .*
  script: app.application

app.py:

from google.appengine.ext import vendor
vendor.add('local')

from future import standard_library
standard_library.install_aliases()

import http.client

then create the virtualenv, run the server:

virtualenv local
source local/bin/activate
pip install future
dev_appserver.py .

and open http://localhost:8080/ in your browser.

i'm on future 0.16.0 and gcloud 184.0.0. details:

$ gcloud info
Google Cloud SDK [184.0.0]

Platform: [Mac OS X, x86_64] ('Darwin', 'hell.local', '17.3.0', 'Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64', 'x86_64', 'i386')
Python Version: [2.7.10 (default, Jul 15 2017, 17:16:57)  [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]]
Python Location: [/Users/ryan/src/gcp-variant-transforms/local/bin/python2]
Site Packages: [Enabled]

Installation Root: [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk]
Installed Components:
  core: [2018.01.05]
  app-engine-python: [1.9.65]
  gcloud: []
  beta: [2017.09.15]
  gsutil: [4.28]
  bq: [2.0.28]
  alpha: [2017.09.15]
System PATH: [...]
Python PATH: [...]
Cloud SDK on PATH: [False]
Kubectl on PATH: [False]
...
git: [git version 2.15.1]
ssh: [OpenSSH_7.6p1, LibreSSL 2.6.2]

[jacobg]

@snarfed When I use the backported urllib (indirectly via python sendgrid client), I get this error in local development:

le "/Users/jacob/workspace/gae/src/sitepackages/prod/python_http_client/client.py", line 172, in _make_request
    return opener.open(request, timeout=timeout)
  File "/Users/jacob/workspace/gae/src/sitepackages/prod/future/backports/urllib/request.py", line 494, in open
    response = self._open(req, data)
  File "/Users/jacob/workspace/gae/src/sitepackages/prod/future/backports/urllib/request.py", line 512, in _open
    '_open', req)
  File "/Users/jacob/workspace/gae/src/sitepackages/prod/future/backports/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/jacob/workspace/gae/src/sitepackages/prod/future/backports/urllib/request.py", line 1326, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Users/jacob/workspace/gae/src/sitepackages/prod/future/backports/urllib/request.py", line 1287, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 13] Permission denied>

It works fine on production app engine.

The http client has this code:

try:
    # Python 3
    import urllib.request as urllib
    from urllib.parse import urlencode
    from urllib.error import HTTPError
except ImportError:
    # Python 2
    import urllib2 as urllib
    from urllib2 import HTTPError
    from urllib import urlencode

If I force it to urllib2, then it works fine.

Have you seen this issue?

[snarfed]

@jacobg heh, yes, i hit that too: https://github.com/snarfed/webutil/commit/f71e1cb479d2218bc3b690384868d90171dc7d80

the backported urllib doesn't support SSL - http://python-future.org/imports.html#aliased-imports - so if you're seeing this on https URLs, that's probably why. try from future.moves.urllib.request import urlopen instead.

[jacobg]

@snarfed haha thanks! This monkeypatch fixes it:

def _fix_sendgrid_http_client():
    # force sendgrid's http client to use urllib2, not future urllib.
    # future urllib doesn't work in local development.
    # https://github.com/PythonCharmers/python-future/issues/320#issuecomment-412174717
    import urllib2 as urllib
    from urllib2 import HTTPError
    from urllib import urlencode
    from python_http_client import client
    client.urllib = urllib
    client.HTTPError = HTTPError
    client.urlencode = urlencode