run_server, other butler commands should support http_proxy

[zlrs]

I have installed prerequisites according to https://google.github.io/clusterfuzz/getting-started/prerequisites/. And it seems I have successfully run the script local/install_deps.bash and I saw the prompt of success:

Installation succeeded!
Please load virtualenv environment by running 'source ENV/bin/activate'.

However, strange error occured when I run $ python butler.py run_server --bootstrap just after the script finished and viturlenv was loaded. And when I requested localhost:9000, I got a http status code 500.

HTTPError()
| HTTPError()
| Traceback (most recent call last):
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 1302, in communicate
|     req.respond()
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 831, in respond
|     self.server.gateway(self).respond()
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 2115, in respond
|     response = self.req.server.wsgi_app(self.env, self.start_response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 292, in __call__
|     return app(environ, start_response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
|     response_body = iter(application(environ, wrapped_start_response))
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 165, in __call__
|     self._flush_logs(response.get('logs', []))
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 308, in _flush_logs
|     apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 97, in MakeSyncCall
|     return stubmap.MakeSyncCall(service, call, request, response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 331, in MakeSyncCall
|     rpc.CheckSuccess()
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 136, in CheckSuccess
|     self._traceback)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 161, in _WaitImpl
|     self.request, self.response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
|     self._MakeRealSyncCall(service, call, request, response)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 248, in _MakeRealSyncCall
|     encoded_response = self._server.Send(self._path, encoded_request)
|   File "/home/ubuntu/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc.py", line 469, in Send
|     f = self.opener.open(req)
|   File "/usr/lib/python2.7/urllib2.py", line 435, in open
|     response = meth(req, response)
|   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
|     'http', request, response, code, msg, hdrs)
|   File "/usr/lib/python2.7/urllib2.py", line 473, in error
|     return self._call_chain(*args)
|   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
|     result = func(*args)
|   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
|     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
| HTTPError: HTTP Error 504: Gateway Time-out
| INFO     2019-03-26 11:37:00,810 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -
.............
| INFO     2019-03-26 11:37:02,928 module.py:861] cron-service: "GET /triage HTTP/1.1" 500 -
| ERROR    2019-03-26 11:37:27,693 base_handler.py:198] HTTP Error 504: Gateway Time-out
...........
| INFO     2019-03-26 11:38:59,858 module.py:861] cron-service: "GET /cleanup HTTP/1.1" 500 -

Here is part of output of python butler.py run_server --bootstrap(also part of the output of ./local/install_deps.bash ). Can anyone tell me why this happened and what should I do next?

| Successfully built google-cloud-profiler
| google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 0.28.1 which is incompatible.
| proto-google-cloud-datastore-v1 0.90.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.3 which is incompatible.
| googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have httplib2 0.11.3 which is incompatible.
| googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll have oauth2client 4.1.3 which is incompatible.
| tensorboard 1.8.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.
| bleach 1.5.0 has requirement html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999, but you'll have html5lib 1.0.1 which is incompatible.

ENV: Ubuntu 18.04 with VMware Pro based on a Windows 10 machine. Google-cloud-sdk version 226.0.0

[inferno-chromium]

Is your machine behind a proxy ? E.g. see https://stackoverflow.com/questions/15761715/gae-dev-appserver-throws-http-504-gateway-timeout is http_proxy or HTTP_PROXY set in your environment ?

[zlrs]

Is your machine behind a proxy ? E.g. see https://stackoverflow.com/questions/15761715/gae-dev-appserver-throws-http-504-gateway-timeout is http_proxy or HTTP_PROXY set in your environment ?

Yes, my machine is behind a proxy. Besides, http_proxy and HTTP_proxy are set in my env variable.

[inferno-chromium]

Yes you need to unset those before starting run_server, this is exactly what is pointed out as the reason in c#2 of https://stackoverflow.com/questions/15761715/gae-dev-appserver-throws-http-504-gateway-timeout

[zlrs]

Yes you need to unset those before starting run_server, this is exactly what is pointed out as the reason in c#2 of https://stackoverflow.com/questions/15761715/gae-dev-appserver-throws-http-504-gateway-timeout

I'd like to reopen this issue because unset http_proxy didn't work for me. As python butler.py run_server --bootstrap goes, at the begining, the script executues pip install. If http_proxy is unset before the script runs, pip install won't work and the scipt sticks.

However, as @inferno-chromium said, in order to solve the 500 error, I have to unset http_proxy. I tried to find the other ways to make pip to use proxy without setting a env variable. But I only found the command line way(not a configuation file).

I have been thinking on it for a long time but couldn't get a clue. Is there any way to solve this?

[inferno-chromium]

We won't have time to debug this anytime soon, so we recommend you debug this and propose a patch.

[wideglide]

I think I have this partially working behind a proxy, using the NO_PROXY=localhost environment variable.

The services appear to start:

Running: bazel run //local:create_gopath (cwd='src')
| Loading:
| Loading: 0 packages loaded
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| INFO: Analyzed target //local:create_gopath (0 packages loaded, 0 targets configured).
| INFO: Found 1 target...
| [0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
| Target //local:create_gopath up-to-date:
|   bazel-bin/local/create_gopath
| INFO: Elapsed time: 0.301s, Critical Path: 0.00s
| INFO: 0 processes.
| INFO: Build completed successfully, 1 total action
| INFO: Running command line: bazel-bin/local/create_gopath local/gopath
| INFO: Build completed successfully, 1 total action
| /home/localuser/.cache/bazel/_bazel_localuser/de65168dfbbea5a8f0776c717b698172/execroot/__main__/bazel-out/k8-fastbuild/bin/local/gopath
Running: python polymer_bundler.py (cwd='local')
| App Engine templates are up to date.
Created symlink: source: /opt/clusterfuzz/local/storage/local_gcs, target /opt/clusterfuzz/src/appengine/local_gcs.
Running: /usr/lib/google-cloud-sdk/bin/dev_appserver.py -A test-clusterfuzz --skip_sdk_update_check=1 --storage_path=local/storage --port=9000 --admin_port=9002 --datastore_emulator_port=9004 --require_indexes=true --log_level=info --dev_appserver_log_level=info --support_datastore_emulator=true --env_var LOCAL_DEVELOPMENT=True --env_var PUBSUB_EMULATOR_HOST=loc
alhost:9006 --env_var LOCAL_GCS_BUCKETS_PATH=local_gcs --env_var LOCAL_GCS_SERVER_HOST=http://localhost:9008 --specified_service_ports=cron-service:9009 src/appengine src/appengine/cron-service.yaml
| INFO     2019-07-06 17:58:43,353 devappserver2.py:224] Using Cloud Datastore Emulator.
| We are gradually rolling out the emulator as the default datastore implementation of dev_appserver.
| If broken, you can temporarily disable it by --support_datastore_emulator=False
| Read the documentation: https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator
| Help us validate that the feature is ready by taking this survey: https://goo.gl/forms/UArIcs8K9CUSCm733
| Report issues at: https://issuetracker.google.com/issues/new?component=187272
|
| INFO     2019-07-06 17:58:43,387 devappserver2.py:278] Skipping SDK update check.
| INFO     2019-07-06 17:58:43,452 datastore_emulator.py:155] Starting Cloud Datastore emulator at: http://localhost:9004
| INFO     2019-07-06 17:58:44,600 datastore_emulator.py:161] Cloud Datastore emulator responded after 1.147306 seconds
| INFO     2019-07-06 17:58:44,601 api_server.py:275] Starting API server at: http://localhost:37807
| INFO     2019-07-06 17:58:44,606 api_server.py:265] Starting gRPC API server at: http://localhost:34775
| INFO     2019-07-06 17:58:44,714 dispatcher.py:256] Starting module "default" running at: http://localhost:9000
| INFO     2019-07-06 17:58:44,820 dispatcher.py:256] Starting module "cron-service" running at: http://localhost:9009
| INFO     2019-07-06 17:58:44,821 admin_server.py:150] Starting admin server at: http://localhost:9002
| INFO     2019-07-06 17:59:33,397 instance.py:294] Instance PID: 4012
| /opt/clusterfuzz/src/appengine/third_party/urllib3/contrib/appengine.py:115: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/latest/reference/urllib3.contrib.html.
|   AppEnginePlatformWarning)
| INFO     2019-07-06 17:59:34,146 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
| INFO     2019-07-06 17:59:34,173 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt

But then I get intermittent HTTP 503 Errors

| INFO     2019-07-06 17:59:34,146 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
| INFO     2019-07-06 17:59:34,173 driver.py:124] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
| HTTPError()
| HTTPError()
| Traceback (most recent call last):
| INFO     2019-07-06 17:59:34,787 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -

| HTTPError: HTTP Error 503: Service Unavailable

| HTTPError: HTTP Error 503: Service Unavailable
| WARNING  2019-07-06 17:59:34,847 tasklets.py:468] suspended generator run_to_queue(query.py:969) raised HTTPError(HTTP Error 503: Service Unavailable)
| WARNING  2019-07-06 17:59:34,847 tasklets.py:468] suspended generator helper(context.py:890) raised HTTPError(HTTP Error 503: Service Unavailable)
| WARNING  2019-07-06 17:59:34,847 tasklets.py:468] suspended generator has_next_async(query.py:1793) raised HTTPError(HTTP Error 503: Service Unavailable)
| WARNING  2019-07-06 17:59:34,847 tasklets.py:468] suspended generator _fetch_page_async(query.py:1380) raised HTTPError(HTTP Error 503: Service Unavailable)
| ERROR    2019-07-06 17:59:34,850 logs.py:329] Error occurred while grouping test cases.

| LogError: Error occurred while grouping test cases.

Any idea why this request would fail to connect and seem to ignore the proxy settings?