search

webrtc / apprtc

appr.tc has been shutdown. Please use the Dockerfile to run your own test/dev instance.
BSD 3-Clause "New" or "Revised" License
4.16k stars 1.37k forks source link

The Analytics change breaks AppRTC test in WebRTC waterfall #21

Closed jiayliu closed 9 years ago

jiayliu commented 9 years ago

https://github.com/GoogleChrome/webrtc/issues/467#event-230065986

phoglund commented 6 hours ago I reverted the analytics change for now. When bringing up the server on the bots after the analytics change:

INFO 2015-02-04 09:56:16,612 devappserver2.py:557] Skipping SDK update check. WARNING 2015-02-04 09:56:16,622 simple_search_stub.py:1002] Could not read search indexes from /tmp/appengine.apprtc.chrome-bot/search_indexes INFO 2015-02-04 09:56:16,623 api_server.py:138] Starting API server at: http://localhost:58086 INFO 2015-02-04 09:56:16,634 dispatcher.py:164] Starting module "default" running at: http://localhost:9999 ERROR 2015-02-04 09:56:17,886 wsgi.py:219] Traceback (most recent call last): File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle handler = config_handle.add_wsgi_middleware(self.LoadHandler()) File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler handler = __import(path[0]) File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in import analytics File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 14, in from oauth2client.client import SignedJwtAssertionCredentials ImportError: cannot import name SignedJwtAssertionCredentials INFO 2015-02-04 09:56:17,891 module.py:593] default: "GET /r/some_room?wshpp=localhost:8089&wstls=false&firefox_fake_device=1 HTTP/1.1" 500 - ERROR 2015-02-04 09:56:17,919 wsgi.py:219] Traceback (most recent call last): File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle handler = config_handle.add_wsgi_middleware(self.LoadHandler()) File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler handler = __import(path[0]) File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in import analytics File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 14, in from oauth2client.client import SignedJwtAssertionCredentials ImportError: cannot import name SignedJwtAssertionCredentials INFO 2015-02-04 09:56:17,923 module.py:593] default: "GET /favicon.ico HTTP/1.1" 500 - BrowserTestBase signal handler received SIGTERM. Backtrace:

0 0x000001645df9 base::debug::StackTrace::StackTrace()

Log from http://chromegw.corp.google.com/i/chromium.webrtc/builders/Linux%20Tester/builds/33276/steps/browser_tests/logs/stdio. phoglund Owner phoglund commented 6 hours ago Where are the oauth2client imports supposed to come from? I can't see it being added or depended upon in the analytics CLs. Are they a part of appengine? We need to figure out why the bots fail to import those when they bring up the server with dev_appserver.py.

(Also, next time you do a merge like this, try to get it right in the branch before merging to master, so I don't have to revert 5 commits with fixes piled on top of each other). phoglund Owner phoglund commented 2 hours ago Strangely enough, the test works when I run on my workstation. I tried deleting my google_appengine copy and webrtc-samples; it still succeeds. Now that is weird. My first thought was that this is due to differences against the bot's PYTHONPATH, but it succeeds.

After digging around a bit, I reckon the code is supposed to come from src/out/webrtc-samples/samples/web/content/apprtc/third_party/oauth2client/client.py. Looking at that code:

if HAS_CRYPTO:

PyOpenSSL and PyCrypto are not prerequisites for oauth2client, so if it is

missing then don't create the SignedJwtAssertionCredentials or the

verify_id_token() method.

class SignedJwtAssertionCredentials(AssertionCredentials):

So the problem is that we don't have PyOpenSSL and PyCrypto? It wouldn't surprise me if the bots' system python do not have those things. Donald, what needs to happen to make that work? Can you pull the crypto stuff into third_party as some kind of build step for AppRTC? As a general thing, we can't install stuff into the bots' system python, so the app needs to pull all its dependencies by itself.

jiayliu commented 9 years ago

It also caused a problem on dev servers:

E 09:46:23.591 <HttpError 404 when requesting https://www.googleapis.com/bigquery/v2/projects/apprtc/datasets/prod/tables/analytics/insertAll?alt=json returned "Not Found: Project apprtc"> Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in call rv = self.handle_exception(request, response, e) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in call rv = self.router.dispatch(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher return route.handler_adapter(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in call return handler.dispatch() File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch return self.handle_exception(e, self.app.debug) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch return method(_args, _kwargs) File "/base/data/home/apps/s~apprtc/3.382002904972735408/join_page.py", line 171, in post allow_room_creation = True) File "/base/data/home/apps/s~apprtc/3.382002904972735408/room.py", line 213, in add_client_to_room host=request.host) File "/base/data/home/apps/s~apprtc/3.382002904972735408/analytics.py", line 110, in report_event analytics.report_event(_args, _kwargs) File "/base/data/home/apps/s~apprtc/3.382002904972735408/analytics.py", line 91, in report_event body=obj).execute() File "/base/data/home/apps/s~apprtc/3.382002904972735408/third_party/oauth2client/util.py", line 132, in positional_wrapper return wrapped(_args, *_kwargs) File "/base/data/home/apps/s~apprtc/3.382002904972735408/third_party/apiclient/http.py", line 723, in execute raise HttpError(resp, content, uri=self.uri) HttpError: <HttpError 404 when requesting https://www.googleapis.com/bigquery/v2/projects/apprtc/datasets/prod/tables/analytics/insertAll?alt=json returned "Not Found: Project apprtc">

I'm going to disable the call for now

milkypostman commented 9 years ago

The app.yaml file specifies that App Engine should pull in the dependencies. I don't understand why this is a problem. When you run the App Engine dev server locally it is supposed to behave exactly like the server. https://github.com/webrtc/apprtc/blob/master/src/app_engine/app.yaml#L33

@phoglund why would the bots seem to be ignoring this?

chuckhays commented 9 years ago

To run python tests on mac (this is in the apprtc-t branch, but also happens in master) I have to add the following to analytics_test.py: import os import sys sys.path.append(os.path.join(os.path.dirname(file), 'third_party')) sys.path.append(os.path.join(os.path.dirname(file), '../third_party'))

I also have to comment out

import analytics in room.py

I also noticed in the apprtc-t branch, apprtc_test.py doesn't have the same changes as master that add analytics tests. Master: https://github.com/webrtc/apprtc/blob/master/src/app_engine/apprtc_test.py apprtc-t: https://github.com/webrtc/apprtc/blob/apprtc-t/src/app_engine/apprtc_test.py

jiayliu commented 9 years ago

The analytics related code in apprtc_test.py is moved to test_utilities.py.

On Wed, Feb 4, 2015 at 11:39 AM, Chuck Hays notifications@github.com wrote:

To run python tests on mac (this is in the apprtc-t branch, but also happens in master) I have to add the following to analytics_test.py: import os import sys sys.path.append(os.path.join(os.path.dirname(file), 'third_party')) sys.path.append(os.path.join(os.path.dirname(file), '../third_party'))

I also have to comment out

import analytics in room.py

I also noticed in the apprtc-t branch, apprtc_test.py doesn't have the same changes as master that add analytics tests. Master: https://github.com/webrtc/apprtc/blob/master/src/app_engine/apprtc_test.py apprtc-t: https://github.com/webrtc/apprtc/blob/apprtc-t/src/app_engine/apprtc_test.py

— Reply to this email directly or view it on GitHub https://github.com/webrtc/apprtc/issues/21#issuecomment-72923358.

Jiayang Liu

jiayliu commented 9 years ago

I've setup the bigquery tables and there is no server error when running on cloud now. We still need to figure out how to fix the tests.

jiayliu commented 9 years ago

BTW, I have to use the project number instead of the project id when calling bigquery.tabledata().insertAll. Not sure how to get project number programmatically.

jiayliu commented 9 years ago

I have https://github.com/webrtc/apprtc/pull/29 to fix the test issues, including the browser test issue and the unit test issue on mac.

jiayliu commented 9 years ago

Fixed

phoglund commented 9 years ago

Donald: yeah, so it would appear it is dev_appserver that somehow provisions these things. It also appears that provision fails. The bots are running rather strictly controlled version of python, so perhaps dev_appserver is looking for something that doesn't exist or is compatible with that version of python. I'll commandeer a bot and see if I can figure something out.

phoglund commented 9 years ago

Hm, well. The python version doesn't exactly look ancient. Bot/Workstation Python 2.7.3/Python 2.7.6 SDK 1.8.3/SDK 1.8.3

I can repro with python google_appengine/dev_appserver.py src/out/webrtc-samples/samples/web/content/apprtc/ curl http://localhost:8080

on the bot.

phoglund commented 9 years ago

Downloading the latest SDK on the bot did not help, so something else is going on.

phoglund commented 9 years ago

Damn it, this is hard to debug. Here's the real error anyway:

import analytics File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 17, in import oauth2client.crypt File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/crypt.py", line 255, in raise ImportError('No encryption library found. Please install either ' ImportError: No encryption library found. Please install either PyOpenSSL, or PyCrypto 2.6 or later

milkypostman commented 9 years ago

@phoglund i wonder if there is something missing on the bots which prevents pycrypto from being installed by dev_appserver.py. I guess I'll have to try more things locally.

phoglund commented 9 years ago

Ah, I was confused by the fact that the oauth2client code isn't pulled from apprtc/third_party/oauth2client, it's pulled from the app engine SDK folder/oauth2client. Keeping digging...

phoglund commented 9 years ago

Ok, so

Traceback (most recent call last): File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) File "/b/build/slave/Linux_Tester/build/google_appengine/google/appengine/runtime/wsgi.py", line 255, in _LoadHandler handler = import(path[0]) File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/apprtc.py", line 26, in import analytics File "/b/build/slave/Linux_Tester/build/src/out/webrtc-samples/samples/web/content/apprtc/analytics.py", line 13, in from oauth2client.appengine import AppAssertionCredentials File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/appengine.py", line 46, in from oauth2client.client import AccessTokenRefreshError File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/client.py", line 43, in from oauth2client import crypt File "/b/build/slave/Linux_Tester/build/google_appengine/oauth2client/crypt.py", line 37, in from OpenSSL import crypto ImportError: No module named OpenSSL

which is really weird, because if I run the system python on the bot I can import it fine. Must be that dev_appserver either launches its own python or hacks sys.path.

milkypostman commented 9 years ago

Well it's weird cause app.yaml specifies SSL and pycrypto.

phoglund commented 9 years ago

Aha, this is where the system gets it on my workstation:

/usr/lib/python2.7/dist-packages/Crypto/init.pyc

So Crypto, whatever that is, or openssh needs to be installed on the system python for this to work with dev_appserver.

phoglund commented 9 years ago

I guess pycrypto isn't as self-contained as you would think, i.e. needs stuff on the system.

phoglund commented 9 years ago

Ha! The bot actually manages to import parts of the Crypto library, but it fails because Crypto.Signature doesn't exist in the bot's version of the package. The Crypto version installed in /usr/lib/python2.7/dist-packages/Crypto/ is new enough on our workstations but not on the bot.

Well, now we know that at least. I'll give some thought on how to solve this tomorrow. I suspect upgrading all the bots somehow is out of the question. There's so much stuff using Python on the bots that an upgrade of anything is a major and risky undertaking. Donald, would it be possible to disable analytics when running under dev_appserver? It may try to load SignedJwtAssertionCredentials, but then it needs to catch the import error and do something appropriate (like shut itself down and not actually do anything when the rest of code calls it). WDYT?

phoglund commented 9 years ago

...Or maybe I can put a more recent copy of the package earlier in the sys.path to force Python to look it up there. The sys.path looks like this when I run:

/usr/local/google/home/phoglund/dev/chrome/src/out/webrtc-samples/samples/web/content/apprtc /usr/local/google/home/phoglund/dev/chrome/google_appengine /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/protorpc /usr/local/google/home/phoglund/dev/chrome/google_appengine /usr/lib/python2.7 /usr/lib/python2.7/lib-dynload /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/jinja2-2.6 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/ssl-2.7 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/pycrypto-2.6 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/markupsafe-0.15 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/setuptools-0.6c11 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webapp2-2.3 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webob-1.1.1 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/yaml-3.10

That could work actually. Well, consider the above comment, that's certainly less hackish.

milkypostman commented 9 years ago

It would be very easy to fix this problem and ignore it on the bots. we simply can import 'oauth2client.appengine' and then use 'AppAssertionCredentials' later based on that import.

On Thu, Feb 5, 2015 at 8:15 AM, phoglund notifications@github.com wrote:

...Or maybe I can put a more recent copy of the package earlier in the sys.path to force Python to look it up there. The sys.path looks like this when I run:

/usr/local/google/home/phoglund/dev/chrome/src/out/webrtc-samples/samples/web/content/apprtc /usr/local/google/home/phoglund/dev/chrome/google_appengine /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/protorpc /usr/local/google/home/phoglund/dev/chrome/google_appengine /usr/lib/python2.7 /usr/lib/python2.7/lib-dynload /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/jinja2-2.6 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/ssl-2.7

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/pycrypto-2.6

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/markupsafe-0.15

/usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/setuptools-0.6c11 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webapp2-2.3 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/webob-1.1.1 /usr/local/google/home/phoglund/dev/chrome/google_appengine/lib/yaml-3.10

That could work actually. Well, consider the above comment, that's certainly less hackish.

— Reply to this email directly or view it on GitHub https://github.com/webrtc/apprtc/issues/21#issuecomment-73073875.