GWT, AJAX, and proxies

[jshslsky]

cf. cens/ohmageServer#314

In English:

1. We have users attempting to access mobilize through a proxy.
2. We can verify that the server returns data for login and the post-login calls.
3. We observed a completely blank page post-login.

The problem trace:

GET https://maps.google.com/maps/api/js?sensor=false 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527Mon Mar 12 14:41:27 GMT-700 2012 edu.ucla.cens.mobilize.client.common.TokenLoginManager
INFO: Initialzing login manager with user name: demo.017 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527Mon Mar 12 14:41:27 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Fetching app config. 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Server response: {"result":"success","data":{"survey_response_privacy_states":["invisible","private","shared"],"application_build":"9f49613","application_name":"mobilize","mobility_enabled":false,"default_campaign_creation_privilege":false,"application_version":"2.10","auth_token_lifetime":1800000,"maximum_parameter_size":5242880,"default_survey_response_sharing_state":"private","maximum_request_size":26214400}} 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527 Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Attempting to fetch user info with parameters: auth_token=4dc87aec-3091-4153-af15 7c4bc5d04e87&client=ohmage-gwt&user_list=demo.017 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Server response: {"result":"success","data":{"demo.017":{"classes":{"urn:class:ca:WashPrep:SP2012:MobilizeCorp":"Mobilize Corp","urn:class:ca:ucla:Mobilize:Teachers:2012":"LAUSD Teachers","urn:class:ca:ucla:Mobilize:Demo:2012":"Demo"},"permissions":{"is_admin":false,"can_create_campaigns":true},"campaign_roles":["supervisor","participant"],"campaigns":{"urn:campaign:ca:ucla:Mobilize:Demo:2012:Snack":"Snack","urn:campaign:ca:ucla:Mobilize:Demo:2012:Media":"Media"},"class_roles":["privileged"]}}} 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527 Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Attempting to fetch campaign list with parameters: auth_token=4dc87aec-3091-4153-af15-7c4bc5d04e87&client=ohmage-gwt&output_format=short 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527 Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.dataaccess.AndWellnessDataService
FINE: Server response: {"result":"success","data":{"urn:campaign:ca:ucla:Mobilize:Demo:2012:Snack":{"user_roles":["participant","supervisor"],"description":"Snack Campaign with developer-sourced data to be used in demos for Mobilize, Spring 2012","name":"Snack","privacy_state":"shared","creation_timestamp":"2011-08-30 21:51:06","running_state":"running"},"urn:campaign:ca:ucla:Mobilize:Demo:2012:Media":{"user_roles":["participant","supervisor"],"description":"Media Campaign with developer-sourced data to be used in demos for Mobilize, Spring 2012","name":"Media","privacy_state":"shared","icon_url":"http://lecs.cs.ucla.edu/mobilize/img/ad_icon.png","creation_timestamp":"2011-08-30 21:51:37","running_state":"running"}},"metadata":{"items":["urn:campaign:ca:ucla:Mobilize:Demo:2012:Snack","urn:campaign:ca:ucla:Mobilize:Demo:2012:Media"],"number_of_results":2}} 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527 Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.MainApp
FINE: Exception in initComponents: (TypeError): Cannot read property 'maps' of undefined 4BBFAEC4E7C440B65BF7FFAA7E46DFB0.cache.html:4527 Mon Mar 12 14:41:28 GMT-700 2012 edu.ucla.cens.mobilize.client.MainApp
SEVERE: Failed to fetch campaign short infos or init app: (TypeError): Cannot read property 'fb' of null

I have checked out the code in-depth and there doesn't appear to be anything going on that is strange. The fact that this problem occurs only with a proxy server in between the client and the ohmage server is telling.

I think the browser is freaking out because of a possible CSRF / CORS resource issue. We are getting undefined or null references where in the non-proxy scenario they just don't occur. It seems like browsers are disallowing access to the JSON returned from the server.

@shlurbee @ewang9 @jeroenooms

[shlurbee]

What's the "cannot read property 'maps'" error? Might it be a problem accessing the google maps api?

[jshslsky]

I think it's a symptom of the same problem we have with ohmage.

In our project planning meeting today, we spent a good 30 minutes discussing this issue. Steve put a proxy on his Mac and we could recreate the whitescreen error consistently. I did not notice the maps issue cropping up.

[shlurbee]

The login page loads, right? You might try commenting out all the tabs in MainApp and reintroducing them one by one to see if a particular tab is causing the problem. Easiest way to do that would probably be to pick one tab like "ExploreData" and search for all occurrences of that string in the file to find related to that tab. (It'll be things like new view, new presenter, set view, etc.)

Do you have the same problem when running the app from Eclipse, or just when compiled?

[jshslsky]

Yes, the login page loads. The problem only occurs when compiled and when there is a proxy server between the GWT client and the ohmage server.

Steve brought up a good point about maps where he was wondering if there was a non-secure call on to the Google Maps API.

[jeroen]

I found the problem. The issue is in the SSL certificates. I am not sure about the proxy at lausd, but Steven's little burp testing proxy replaces all HTTPS with a self-signed certificate. This looks highly suspicious to a browser, because it smells like a MITM attack.

This is why you are getting the big warning in firefox, and you have to 'confirm the security exception for this server' when visiting https://mdev. However, the frontend actually talks to 4 other https servers (other than mdev) which it will block until you manually added these individual security exceptions:

https://maps.gstatic.com
https://www.google.com
https://maps.google.com
https://maps.googleapis.com

Unfortunately, because these requests happen in the background, the browser does not display the big 'warning this page has an insecure certificate' page. It just blocks it.

To test try the following:

• enable the https proxy
• flush the cache
• visit all of the servers mentioned above manually by copy pasting the url in your browser, and add security exceptions for all of them.
• login at mdev.

Now the page works again, because the frontend won't block the ssl to the google servers anymore. However I am not sure if we can really fix this, because it is really bad behaviour of the proxy if it intercepts https and replaces it with it's own certificate. It goes against the entire system of only trusting certificates signed by a certificate authority.

Best fix I can think of is trying to use the non-ssl google api's where possible. The ones that only have SSL we might have to proxy through ohmage if we want to deal with these bad proxies.

[stevenolen]

What if we just asked users to visit maps.google.com when they had they trouble and confirm the security exception? It sounds like the issue at LAUSD only showing on certain systems is because some systems have already visited google maps and confirmed the exception?

Maybe we can talk to LAUSD about how they handle situations like this -- I'm sure it's come up before.

[jeroen]

Yeah probably. I think it is mostly https://maps.googleapis.com that makes things block. They probably proxy https because they want to filter naughty sites, etc. But it goes against the entire idea of https if you just start intercepting connections and sign them with your own certificate like nothing happened.

If they really want to do this, they should import the proxy's SSL certificate as a root certificate authority in all of their browsers. Then it will pass. But I highly doubt they know what they're doing.

[stevenolen]

They do actually offer this, but as far as I know only for teachers on teacher machines...the lab machines are highly under-managed in many cases. On Apr 26, 2012 8:30 PM, "jeroenooms" < reply@reply.github.com> wrote:

Yeah probably. They probably do it because they want to filter naughty sites, etc. But it goes against the entire idea of https of course if you just start intercepting connections and sign them with your own certificate like nothing happened.

If they really want to do this, they should import the proxy's SSL certificate as a root certificate authority in all of their browsers. Then it will pass. But I highly doubt they know what they're doing.

---

Reply to this email directly or view it on GitHub: https://github.com/cens/ohmageFrontEnd/issues/247#issuecomment-5372852

[jshslsky]

ok, here is a fix for this in 2.10.

Add a key to the preference table that can be served up from config/read. The key can be something like forceHttpForMobilizeProxy although at this point I'd like to add some expletives to the key. If the value for the key is true, the front end can default to using plain HTTP for calls to Google Maps. In later versions of ohmage, we will probably encounter the same issue with the secure recaptcha calls.

[stevenolen]

Results from the field...

I was at a school today and had investigated this. The suggestion the @jeroenooms offered (visiting each site and allowing the exceptions) seems to work only in Firefox...Safari didn't want to save the exceptions and Chrome completely rejected me from visiting https://maps.google.com, with no exception allowed. From here I talked to the teacher (who is the lab manager, as is the case for a number of classes we've visited). His lab was brand new 27" imacs and he had not yet stepped through the root cert process (something that perhaps he was supposed to do?).

He followed this process on one system and it began working perfectly - as expected. As soon as the client accepts the LAUSD cert as valid, the issue goes away. http://servicedesk.lausd.net/cert_filter.htm

The issue is, the root cert needs to be installed on each machine, something I'm not sure the teachers will do. It looks like some systems already have this installed (we can presume this is the reason why only 10% of systems had issues).

Knowing this, do you feel it worth it to implement a change as you've suggested? If everyone thinks a change of that is for the worse, and something we need to manage and maintain differently going forward, we can test the waters and get the root cert installed where it isn't. It will however, be tough to get that done in the middle of the deployment.

[jshslsky]

Hongsuda and I talked and we decided that the new preference key was the best solution because it will mean less work for you and the teachers. :)

[stevenolen]

Got it...thank you, thank you! I'm notifying the teachers that a change is on the way -- but that if they have continued trouble, or need a fix before the fix (which may already be in place, but I promise I wont offer, since we haven't had a chance to test widely), the root cert is available. I think that as soon as they are aware this is a result of the LAUSD network, they will be more than understanding.

[stevenolen]

Do we have an ETA on this fix? I know that at least one teacher is likely to be done using the frontend early this week.

[jshslsky]

We can do the server fix, but we are constrained on front-end resources at the moment.

For now, I would say that either the certificate for the proxy server needs to be added to the browsers on the computers will the "white screen" problem. Or you can just ask the teacher to navigate to Google Maps over SSL for any problem computers.

[jshslsky]

@stevenolen on 4/24 I upgraded adev to start work for 2.12 development. In order to test this, we will need to replace /opt/aw with a backup copy. Is this currently possible with the backups you set up?

[stevenolen]

@joshuaselsky adev does not have the backup method set up yet because of limited space. Do you need a backup prior to today, or just one to use after you are finished testing? I can try to get the space we need today...at a school at the moment though.

[jshslsky]

Don't worry about it; John and I can figure out how best to test once we have a fix. It would've been easier to recreate the old 2.10 from a backup because 2.12 is quite different, but there are other options.

[stevenolen]

Can we spawn another vm and restore an lausd backup to it? Does that make sense? LAUSD has backups of /opt/aw et all. Or maybe it would be better to clone the lausd vm to another container? I really don't know if openvz can be used for something like that...can't wait until we have some boxes I can actually manage...

[jshslsky]

:) That's a very good idea about cloning the LAUSD VM.

[jshslsky]

Here's the fix:

In MobilizeWeb.gwt.xml, change the hardcoded google https URLs to use http. This should resolve the white screen issue, but users will see a message like "You are on a page served up by a secure connection, but it contains non-secure elements".

We will not have to perform any server-side changes for this issue, which is good because the previous solution involved exposing a big security hole (it enabled man-in-the-middle attacks).

Once we get the VM cloned, I can make this change in the front end and then we can test using Steve's proxy test.

[stevenolen]

@joshuaselsky this works well (except the security warning, but I can explain that to teachers). What will the fix entail on the deployment server...any downtime or just a quick restart of tomcat? We can discuss when this should happen.

[jshslsky]

Good to hear! The deployment will be a quick restart of Tomcat with a new version of the front end.

[jshslsky]

Closing as this has been successfully tested and deployed.