iitc-project / ingress-intel-total-conversion

ingress.com/intel total conversion user script with some new features. Should allow easier extension of the intel map.
http://iitc.jonatkins.com/
ISC License
991 stars 553 forks source link

Obfuscated parameters #518

Closed KeithMoyer closed 11 years ago

KeithMoyer commented 11 years ago

It seems the V4 parameters are now obfuscated. The obfuscation seems to be the same for everyone though. Example {"5rc0561uauf6x13u":[{"bzeizowtguoyrrtt":"13_1886_3169","7qej3eqg4sefuaac":37649034,"yqegc976egk5q9vo":-97119141,"2odsgh99ix9bbtsb":37683820,"g9jess8dwa2j8pwi":-97075195,"y853tux9h7cb6xp3":"13_1886_3169"}],"uuo2zqhhy5bw80fu":"dashboard.getThinnedEntitiesV4"}

But, I'm guessing they did it so they can change them periodically on a whim...

KeithMoyer commented 11 years ago

And, it's now changed. So, they probably aren't just changing it on a whim. They're changing it periodically.

I guess IITC would have to periodically download gen_dashboard.js and parse it to see what the new param values are. Or, succumb to their obvious desire to lock us out...

jonatkins commented 11 years ago

Just saw this myself, and posted a message to the users on G+

I can't see them changing it too often - users would have to force-reload to get a new gen_dashboard loaded to parse it, yes? Perhaps they've upgraded their js obfuscation tools to include the client/server json protocol?

jonatkins commented 11 years ago

Interesting... looks like the gen_dashboard has some kind of 'decrypt' that must happen - they still reference entities with sensible names such as 'portalV2', 'resonatorArray', etc

KeithMoyer commented 11 years ago

It's already changed in the 5 minutes I was looking at it. :-P I'm guessing they'd accept old values for a certain amount of time to avoid that problem...

jonatkins commented 11 years ago

OK, the parameters sent TO the server are encrypted in the source. e.g. boundsParamsList changed to n27qzc8389kgakyv, then later to 5rc0561uauf6x13u

I can see two possibilities at the moment

  1. they run the codebase through an obsfucation tool, and have now turn on the option to include the client/server json in the obsfucation. Every time they deploy a new build the values get changed. It's pretty common after a new release goes live for multiple further updates to be sent later.
  2. there's intentional encryption, and some kind of decryption in the code. The parameters sent to the server could be used there to decide how to encrypt for the response

I'm still looking through the changes to the code to figure this all out. I'm glad I set up that cron job that emails me the diffs every time the JS changes....

jonatkins commented 11 years ago

For reference, here's the diffs between the first and second encrypted gen_dashboard changes I saw. (the diffs between the original unencrypted json and first encrypted one are much larger, due to lots of obfuscated function/variable names changing in the code)

The cron job is just a wget + diff of the file, and at this time ran once per hour. It's not logged into the intel site like a user would be. I've increased the frequency to catch additional changes...

--- gen_dashboard.js.orig   2013-08-30 23:09:02.313764430 +0100
+++ gen_dashboard.js    2013-08-31 00:09:01.691506222 +0100
@@ -2120,7 +2120,7 @@
   qb(te(this, a), b.target)
 };
 se.prototype.na = function(a, b, c, d, f) {
-  b["4kr3ofeptwgary2j"] = a;
+  b.uuo2zqhhy5bw80fu = a;
   _gaq.push(["_trackEvent", "RPC", a]);
   var e = new he, g = u(this.Fe, this, a), c = u(this.Ce, this, a, c), f = qa(ue, a, f), j = {};
   j.complete = g;
@@ -2153,9 +2153,9 @@
   var d = [], f = c.length > 8 ? 4 : 1;
   F(c, function(a, b) {
     for(var c = b % f;d.length <= c;) {
-      d.push({n27qzc8389kgakyv:[]})
+      d.push({"5rc0561uauf6x13u":[]})
     }
-    d[c].n27qzc8389kgakyv.push({"39031qie1i4aq563":a.ed, pg98bwox95ly0ouu:Math.round(a.za.bounds.sw.lat() * 1E6), eib1bkq8znpwr0g7:Math.round(a.za.bounds.sw.lng() * 1E6), ilfap961rwdybv63:Math.round(a.za.bounds.ne.lat() * 1E6), lpf7m1ifx0ieouzq:Math.round(a.za.bounds.ne.lng() * 1E6), "2ewujgywmum1yp49":a.U, bgxibcomzoto63sn:a.ed})
+    d[c]["5rc0561uauf6x13u"].push({bzeizowtguoyrrtt:a.ed, "7qej3eqg4sefuaac":Math.round(a.za.bounds.sw.lat() * 1E6), yqegc976egk5q9vo:Math.round(a.za.bounds.sw.lng() * 1E6), "2odsgh99ix9bbtsb":Math.round(a.za.bounds.ne.lat() * 1E6), g9jess8dwa2j8pwi:Math.round(a.za.bounds.ne.lng() * 1E6), "604f34zcu9zna0a5":a.U, y853tux9h7cb6xp3:a.ed})
   });
   var e = 10 * d.length;
   F(d, function(a) {
@@ -2173,13 +2173,13 @@
   }
   f = f || -1;
   e = e || -1;
-  b = {tmb0vgxgp5grsnhp:b, pg98bwox95ly0ouu:Math.round(d.bounds.sw.lat() * 1E6), eib1bkq8znpwr0g7:Math.round(d.bounds.sw.lng() * 1E6), ilfap961rwdybv63:Math.round(d.bounds.ne.lat() * 1E6), lpf7m1ifx0ieouzq:Math.round(d.bounds.ne.lng() * 1E6), hljqffkpwlx0vtjt:f, sw317giy6x2xj9zm:e};
-  c && (b["0dvtbatgzcfccchh"] = k);
-  f > -1 && (b.f6u1iqep9s2lc5y5 = k);
+  b = {sfv5i7l6ouljz8vf:b, "7qej3eqg4sefuaac":Math.round(d.bounds.sw.lat() * 1E6), yqegc976egk5q9vo:Math.round(d.bounds.sw.lng() * 1E6), "2odsgh99ix9bbtsb":Math.round(d.bounds.ne.lat() * 1E6), g9jess8dwa2j8pwi:Math.round(d.bounds.ne.lng() * 1E6), y3g07dbnw6sklloj:f, "3pdl28aa27xvyhke":e};
+  c && (b["37okcr7gvd5yn2lj"] = k);
+  f > -1 && (b.iimftkq7flskwrx9 = k);
   this.na("dashboard.getPaginatedPlextsV2", b, a, 12)
 };
 function xe(a, b, c) {
-  a.na("dashboard.getPlayersByGuids", {pusjrhxxtyp5nois:b}, c)
+  a.na("dashboard.getPlayersByGuids", {xp1pl2jm5hrh3bna:b}, c)
 }
 ;
 // Input 68
@@ -2524,7 +2524,7 @@
     this.Oc.value = "";
     this.rc.innerHTML = "sending invite to " + a + "...";
     var b = u(this.Ae, this, a);
-    R.f().M.na("dashboard.sendInviteEmail", {cltkepgqkepfsyaq:a}, b);
+    R.f().M.na("dashboard.sendInviteEmail", {"2pyrttrp3gh38mmu":a}, b);
     _gaq.push(["_trackEvent", "Action", "Send invite"])
   }else {
     this.rc.innerHTML = "invalid email"
@@ -4140,8 +4140,8 @@
   Ah(this, u(this.nd, this, a, b, k, q), a, b, i, this.D[a][b].g.length < 1 ? -1 : this.D[a][b].g[0].U)
 };
 r.Je = function(a) {
-  var b = u(this.refresh, this, k), c = this.n.getCenter(), d = Math.floor(c.lat() * 1E6), f = Math.floor(c.lng() * 1E6), c = this.M, a = {q0d6n7t1801bb6xu:a, "5ygbhpxfnt1u9e4t":d, ak6twnljwwcgd7cj:f};
-  this.A == "faction" && (a["0dvtbatgzcfccchh"] = k);
+  var b = u(this.refresh, this, k), c = this.n.getCenter(), d = Math.floor(c.lat() * 1E6), f = Math.floor(c.lng() * 1E6), c = this.M, a = {zz54435vfc57nlg9:a, cyltxjod3jhxgj8q:d, h9whcgcz6kpqkz80:f};
+  this.A == "faction" && (a["37okcr7gvd5yn2lj"] = k);
   c.na("dashboard.sendPlext", a, b, 5)
 };
 r.reload = function(a) {
jonatkins commented 11 years ago

.. and as soon as I post that, I spot yet another change to the file. For reference, diffs again

--- gen_dashboard.js.orig   2013-08-31 00:09:01.691506222 +0100
+++ gen_dashboard.js    2013-08-31 00:29:01.793441935 +0100
@@ -2120,7 +2120,7 @@
   qb(te(this, a), b.target)
 };
 se.prototype.na = function(a, b, c, d, f) {
-  b.uuo2zqhhy5bw80fu = a;
+  b["4kr3ofeptwgary2j"] = a;
   _gaq.push(["_trackEvent", "RPC", a]);
   var e = new he, g = u(this.Fe, this, a), c = u(this.Ce, this, a, c), f = qa(ue, a, f), j = {};
   j.complete = g;
@@ -2153,9 +2153,9 @@
   var d = [], f = c.length > 8 ? 4 : 1;
   F(c, function(a, b) {
     for(var c = b % f;d.length <= c;) {
-      d.push({"5rc0561uauf6x13u":[]})
+      d.push({n27qzc8389kgakyv:[]})
     }
-    d[c]["5rc0561uauf6x13u"].push({bzeizowtguoyrrtt:a.ed, "7qej3eqg4sefuaac":Math.round(a.za.bounds.sw.lat() * 1E6), yqegc976egk5q9vo:Math.round(a.za.bounds.sw.lng() * 1E6), "2odsgh99ix9bbtsb":Math.round(a.za.bounds.ne.lat() * 1E6), g9jess8dwa2j8pwi:Math.round(a.za.bounds.ne.lng() * 1E6), "604f34zcu9zna0a5":a.U, y853tux9h7cb6xp3:a.ed})
+    d[c].n27qzc8389kgakyv.push({"39031qie1i4aq563":a.ed, pg98bwox95ly0ouu:Math.round(a.za.bounds.sw.lat() * 1E6), eib1bkq8znpwr0g7:Math.round(a.za.bounds.sw.lng() * 1E6), ilfap961rwdybv63:Math.round(a.za.bounds.ne.lat() * 1E6), lpf7m1ifx0ieouzq:Math.round(a.za.bounds.ne.lng() * 1E6), "2ewujgywmum1yp49":a.U, bgxibcomzoto63sn:a.ed})
   });
   var e = 10 * d.length;
   F(d, function(a) {
@@ -2173,13 +2173,13 @@
   }
   f = f || -1;
   e = e || -1;
-  b = {sfv5i7l6ouljz8vf:b, "7qej3eqg4sefuaac":Math.round(d.bounds.sw.lat() * 1E6), yqegc976egk5q9vo:Math.round(d.bounds.sw.lng() * 1E6), "2odsgh99ix9bbtsb":Math.round(d.bounds.ne.lat() * 1E6), g9jess8dwa2j8pwi:Math.round(d.bounds.ne.lng() * 1E6), y3g07dbnw6sklloj:f, "3pdl28aa27xvyhke":e};
-  c && (b["37okcr7gvd5yn2lj"] = k);
-  f > -1 && (b.iimftkq7flskwrx9 = k);
+  b = {tmb0vgxgp5grsnhp:b, pg98bwox95ly0ouu:Math.round(d.bounds.sw.lat() * 1E6), eib1bkq8znpwr0g7:Math.round(d.bounds.sw.lng() * 1E6), ilfap961rwdybv63:Math.round(d.bounds.ne.lat() * 1E6), lpf7m1ifx0ieouzq:Math.round(d.bounds.ne.lng() * 1E6), hljqffkpwlx0vtjt:f, sw317giy6x2xj9zm:e};
+  c && (b["0dvtbatgzcfccchh"] = k);
+  f > -1 && (b.f6u1iqep9s2lc5y5 = k);
   this.na("dashboard.getPaginatedPlextsV2", b, a, 12)
 };
 function xe(a, b, c) {
-  a.na("dashboard.getPlayersByGuids", {xp1pl2jm5hrh3bna:b}, c)
+  a.na("dashboard.getPlayersByGuids", {pusjrhxxtyp5nois:b}, c)
 }
 ;
 // Input 68
@@ -2524,7 +2524,7 @@
     this.Oc.value = "";
     this.rc.innerHTML = "sending invite to " + a + "...";
     var b = u(this.Ae, this, a);
-    R.f().M.na("dashboard.sendInviteEmail", {"2pyrttrp3gh38mmu":a}, b);
+    R.f().M.na("dashboard.sendInviteEmail", {cltkepgqkepfsyaq:a}, b);
     _gaq.push(["_trackEvent", "Action", "Send invite"])
   }else {
     this.rc.innerHTML = "invalid email"
@@ -4140,8 +4140,8 @@
   Ah(this, u(this.nd, this, a, b, k, q), a, b, i, this.D[a][b].g.length < 1 ? -1 : this.D[a][b].g[0].U)
 };
 r.Je = function(a) {
-  var b = u(this.refresh, this, k), c = this.n.getCenter(), d = Math.floor(c.lat() * 1E6), f = Math.floor(c.lng() * 1E6), c = this.M, a = {zz54435vfc57nlg9:a, cyltxjod3jhxgj8q:d, h9whcgcz6kpqkz80:f};
-  this.A == "faction" && (a["37okcr7gvd5yn2lj"] = k);
+  var b = u(this.refresh, this, k), c = this.n.getCenter(), d = Math.floor(c.lat() * 1E6), f = Math.floor(c.lng() * 1E6), c = this.M, a = {q0d6n7t1801bb6xu:a, "5ygbhpxfnt1u9e4t":d, ak6twnljwwcgd7cj:f};
+  this.A == "faction" && (a["0dvtbatgzcfccchh"] = k);
   c.na("dashboard.sendPlext", a, b, 5)
 };
 r.reload = function(a) {
ddellspe commented 11 years ago

how does this compare on those lines to what was done before? Were the obfuscated strings a common word or trigger?

peta commented 11 years ago

I wrote and started a cron job that fetches gen_dashboard.js every minute and uses the timestamp as filename. Tomorrow when I wake up I will do some batch analysis. Let's hope that I can discover enough similarities to solve this issue.

jonatkins commented 11 years ago

I may have been getting confused here. It's only the request parameters that are obfuscated, and not the response ones, right? I first thought it was both...

ddellspe commented 11 years ago

@jonatkins that's what it looks like, but there's no guarantee that they'll stay the same. There's got to be some sort of connection between the original phrases and these new obfuscated ones

jonatkins commented 11 years ago

@dirtydave0221 as for previous obsfucations - the stock site javascript has always been processed in some way, with all the variable/function names reduced to meaningless letters. This makes it tricky to understand what their code does. However, before now, the actual json protocol between browser/server has been completely in the clear, nothing hidden, no unreadable strings.

ddellspe commented 11 years ago

@jonatkins are there any additional cookies that are available for scouring that you know of?

KeithMoyer commented 11 years ago

@jonatkins Exactly. Just the request. Wanted to respond as soon as your reply came across, but I was out turning green to blue... :-)

So, I don't think we need to try and work out the obfuscation generation. If we just pull the new values out of the source and use them, we're good.

For clarification: I don't think there's any smarts in the stock intel code to "understand" the strings, so I don't think we should try to add any to that regard, either.

jonatkins commented 11 years ago

The problem with that is if the stock site code changes, and re-obsfucation of the code changes function/variable names, how do we go about finding these labels?

So far my own cron job has only seen a couple of different values, with a few changes back/forward between them. Odd.

Perhaps best to wait a few hours, let things settle down and them get into a state where they're not pushing new site builds out - see where we stand then.

ccjon commented 11 years ago

Not sure if it matters but the obfuscation may be the result of running through the Closure compiler... here is an info sheet for preparing Dojo source files for the compiler, as an example:

http://dojo-toolkit.33424.n3.nabble.com/file/n2636749/Using_the_Dojo_Toolkit_with_the_Closure_Compiler.pdf

...if this is true there may be some keys in the Closure docs for getting references resolved.

jonatkins commented 11 years ago

I've been monitoring changes to the stock gen_dashboard.js - and it seems to be alternating between two versions, with different sets of request parameter munging

The list of renamed parameters for one set is (assuming I've done it right) method: '4kr3ofeptwgary2j', boundsParamsList: 'n27qzc8389kgakyv', id: '39031qie1i4aq563', minLatE6: 'pg98bwox95ly0ouu', minLngE6: 'eib1bkq8znpwr0g7', maxLatE6: 'ilfap961rwdybv63', maxLngE6: 'lpf7m1ifx0ieouzq', timestampMs: '2ewujgywmum1yp49', qk: 'bgxibcomzoto63sn', desiredNumItems: 'tmb0vgxgp5grsnhp', minTimestampMs: 'hljqffkpwlx0vtjt', maxTimestampMs: 'sw317giy6x2xj9zm', guids: 'pusjrhxxtyp5nois', inviteeEmailAddress: 'cltkepgqkepfsyaq', message: 'q0d6n7t1801bb6xu', latE6: '5ygbhpxfnt1u9e4t', lngE6: 'ak6twnljwwcgd7cj', factionOnly: '0dvtbatgzcfccchh'

The same parameter used in more than one location (e.g. minLatE6 - in getThinnedEntitiesV4 and getPaginatedPlextsV2) are replaced with the same random-looking string.

I'll go through and work out the other set of parameter mungings too, and then work on some code that can rename the parameters in the ajax request before sending it to the server.

The question is - does the server accept either set of mungings? Is there something - cookies? - it uses to decide if one or another set should be used? How often will Niantic add new sets/retire old ones?

All unknowns at the moment... I wish I had an (unofficial) communication path to their developers here, find out for certain if they're really trying to stop IITC, or if we're just a casualty in changes they've made for another reason.

KeithMoyer commented 11 years ago

Well, you can get gen_dashboard.js without any cookies or an existing session, so it can't rely on that. So, that's good news. :-)

KeithMoyer commented 11 years ago

Another interesting tidbit: it's not just the obfuscation mappings that change, the actual code changes, too: It switches between referencing the bounds params list as a table entry (obj["blah"]) and a member (obj.blah). Also, the declaration of the parameter names changes between being quoted and not quoted (some are quoted in each version, but not the same ones).

So, the obfuscation changes being the result of different passes through a obfuscator/optimizer makes sense.

jonatkins commented 11 years ago

Those obj["prop"] vs obj.prop are just the differences required for properties that do/don't start with a number. I bet in their code it's all "..." style, but the obfuscater also optimises property access to . style where it can.

Also, I didn't spot it originally, but the URL to request from changed from /rpc/dashboard.method to /r/dashboard.method

For reference, the second parameter munge set is method: 'uuo2zqhhy5bw80fu', boundsParamsList: '5rc0561uauf6x13u', id: 'bzeizowtguoyrrtt', minLatE6: '7qej3eqg4sefuaac', minLngE6: 'yqegc976egk5q9vo', maxLatE6: '2odsgh99ix9bbtsb', maxLngE6: 'g9jess8dwa2j8pwi', timestampMs: '604f34zcu9zna0a5', qk: 'y853tux9h7cb6xp3', desiredNumItems: 'sfv5i7l6ouljz8vf', minTimestampMs: 'y3g07dbnw6sklloj', maxTimestampMs: '3pdl28aa27xvyhke', guids: 'xp1pl2jm5hrh3bna', inviteeEmailAddress: '2pyrttrp3gh38mmu', message: 'zz54435vfc57nlg9', latE6: 'cyltxjod3jhxgj8q', lngE6: 'h9whcgcz6kpqkz80', factionOnly: '37okcr7gvd5yn2lj'

and, using this one (but not the first), and the correct URL... I have it working! wooo! :)

jonatkins commented 11 years ago

Well, this change works on desktop (Chrome). Mobile isn't working though - and that's trickier to debug.

I'll try adding code to cycle between the different parameter munge sets on API errors - fingers crossed that might fix things.

KeithMoyer commented 11 years ago

Rather than reactively cycling between, what would you think about proactively detecting (once) which set is the right one?

Something along these lines in place of "var activeMunge = requestMunges[0];" (be gentle, I'm not a JavaScript guy...).

  if (self.activeMunge === undefined)
  {
    for (var m in window) {        
      if (typeof window[m] != "function" || !window.hasOwnProperty(m))
        continue;
      if (m == 'requestDataMunge')
        continue;
      for (var i=0; i < requestMunges.length; ++i) {
        if (window[m].toString().indexOf(requestMunges['id']) >= 0) {
          self.activeMunge = requestMunges[i];
          console.log("Found munge match with set #"+i);
          break;
        }
      }
      if (self.activeMunge !== undefined)
        break;
    }
    if (self.activeMunge === undefined) {
      console.log("Couldn't find munge match! Trying set #0 anyway...");
      self.activeMunge = requestMunges[0];
    }
  }
KeithMoyer commented 11 years ago

Bonus: the above work on both desktop and mobile.

In patch form:

From 70790ff6cdb5d63f172bbd3c961e1633fb0c6128 Mon Sep 17 00:00:00 2001
From: Keith Moyer <ingress@keithmoyer.com>
Date: Sat, 31 Aug 2013 02:16:06 -0500
Subject: [PATCH] Auto-detect proper munge option

---
 code/utils_misc.js |   23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/code/utils_misc.js b/code/utils_misc.js
index a68ea85..5356101 100644
--- a/code/utils_misc.js
+++ b/code/utils_misc.js
@@ -142,7 +142,28 @@ window.requestDataMunge = function(data) {
     },
   ];

-  var activeMunge = requestMunges[1];
+  if (self.activeMunge === undefined)
+  {
+    for (var m in window) {        
+      if (typeof window[m] != "function" || !window.hasOwnProperty(m))
+        continue;
+      if (m == 'requestDataMunge')
+        continue;
+      for (var i=0; i < requestMunges.length; ++i) {
+        if (window[m].toString().indexOf(requestMunges['id']) >= 0) {
+          self.activeMunge = requestMunges[i];
+          console.log("Found munge match with set #"+i);
+          break;
+        }
+      }
+      if (self.activeMunge !== undefined)
+        break;
+    }
+    if (self.activeMunge === undefined) {
+      console.log("Couldn't find munge match! Trying set #0 anyway...");
+      self.activeMunge = requestMunges[0];
+    }
+  }

   function munge(obj) {
     if (Object.prototype.toString.call(obj) === '[object Array]') {
-- 
1.7.10.4
peta commented 11 years ago

Latest test build for Desktop works! :+1:

leCradle commented 11 years ago

Compiled the mobile app with 5b6ed4ce50524584e39c7b1c93168f4729e4ab3b and it works flawlessly.

vogon1 commented 11 years ago

Whatever I try with the test build - it does not work. @peta did you do anything extra besides installing the beta version from the iitc website?

EDIT - it works! When adding the patch from @SgtPepperKSU I am missing resonators though when fully zoomed in.

peta commented 11 years ago

@vogon1 No. Only uninstalled the old one before ... but don't know if this makes any difference.

realchrisolin commented 11 years ago

@SgtPepperKSU Something must have changed in the last 8 hours. I still get errors, even with your patch on both the mobile and desktop versions.

jonatkins commented 11 years ago

I had already started code along these lines last night, so didn't merge your patch as it stands.

Also @leCradle - I had to modify the mobile app so it no longer blocks loading of the stock gen_dashboard.js - otherwise this searching code has nothing to match against. It likely worked for you as it falls back to set 0, which was the opposite of the set 1 default I committed last night.

jonatkins commented 11 years ago

043690f99b03ea7ef39ed11ee25d28e90f5cd9af and 61177c4cda490ac883d9a63b10e48c3c14807939 committed - should get things working on desktop and mobile.

It correctly detects which munge set is in use for me, both on desktop and mobile. Test builds on the website built to include these changes.

leCradle commented 11 years ago

@jonatkins sounds reasonable. Did you figured out when set 0 or set 1 is used? Because I and all my german ingress friends who tested it were good with set 0 the whole day. Maybe it's location based? Just guessing...

jonatkins commented 11 years ago

No idea on how they decide if set 0 or 1 should be used.

My guess is that something from the user's cookies is used to calculate it.

Whatever logic they use - it doesn't even seem to work right for the stock site!

From last night until 5 minutes ago, my gen_dashboard.js (on desktop) had contained the 'set 1' munges. And this worked for me, which is why the first change I committed used this by default.

However, the www.ingress.com site is now sending me a gen_dashboard with the munge set 0

But.. something hasn't told the RPC interface that set 0 should be used - so no neither IITC or the stock intel site is working for me

Either they have someone there actively monitoring things, and making changes to break things.. or - and more likely - it's just badly broken.

I think, perhaps, that just catching the server response of {"error": "invalid method params"} and cycling the munge set in use might be the best option.

KeithMoyer commented 11 years ago

Or just set it back to undefined so it's redetected when that's seen. (EDIT: never mind on that. You'd also have to reload the gen_dashboard.js. And, since there's currently only two known sets, alternating it would make more sense).

I'm guessing it has to do with the server on which your request gets processed, which might be affected by location, rather than directly based on location or cookies. As I mentioned before, you can get the gen_dashboard.js file without any cookies and it still alternates over time.

jonatkins commented 11 years ago

ok, it's not the cleanest way of doing things, but 8448b71faea63fde4ec37d3f624a641a0fde8d70 is yet another attempt at fixing this.

to easily detect+correct invalid munging needs a request that

  1. is not called more than once in parallel
  2. has simple (or no) parameters
  3. happens early

the game score update process matches all of these, so I've thrown in some code there. It fixes things again for me on desktop

Teleken commented 11 years ago

I haven't been able to help, but thank you to everyone who did. And why is Niantic wasting their time on this? So frustrating.

tony1223 commented 11 years ago

I think the biggest reason is for intel parsers, they was made a lot of mechanisms to stop parser.

In past , we could use zoom:-1 to get all portals(ignoring level) in V2, haven't found the change in V3 since it just alive for 2 week,then they limit tiles in a response in V4 to 4. (in past, it's 20.)

And they start to blocking IPs for those making too many requests, I don't think that's for IITC , but it should be parser. It could be a huge effort for server to serve all parsers.

tony1223 commented 11 years ago

If we couldn't get all set, why not learns the set from gen_dashboard ? :) Made a ajax request to get the script source , find those parameters in the js then use it.

like parsing follow code. (method name will change , but still something not changing here... :) )

function we(a, b, c) { var d = [], f = c.length > 8 ? 4 : 1; F(c, function(a, b) { for(var c = b % f;d.length <= c;) { d.push({"5rc0561uauf6x13u":[]}) } d[c]["5rc0561uauf6x13u"].push({bzeizowtguoyrrtt:a.ed, "7qej3eqg4sefuaac":Math.round(a.za.bounds.sw.lat() * 1E6), yqegc976egk5q9vo:Math.round(a.za.bounds.sw.lng() * 1E6), "2odsgh99ix9bbtsb":Math.round(a.za.bounds.ne.lat() * 1E6), g9jess8dwa2j8pwi:Math.round(a.za.bounds.ne.lng() * 1E6), "604f34zcu9zna0a5":a.U, y853tux9h7cb6xp3:a.ed}) }); var e = 10 * d.length; F(d, function(a) { s.setTimeout(u(this.na, this, "dashboard.getThinnedEntitiesV4", a, b, e), 1E3) }, a) }

tony1223 commented 11 years ago

I was writing a parser for parsing JavaScript for parameters this afternoon, and I found a issue and I fixed it again.

Let's see if it could work properly in next few days... :Q

jonatkins commented 11 years ago

FYI, I've just pushed out a new stable IITC build.

Let's see what the next move from Niantic is...

jonatkins commented 11 years ago

Also, just checking my cron job monitoring gen_dashboard changes - has stopped alternating between two versions in the last 12 or so hours, now sticking with munge set 0

jonatkins commented 11 years ago

FYI to anyone monitoring this issue - there was a brief period earlier where another set of munges went live - and this time they also added similar munging to the value of the 'method' parameter.

This gave me a chance to understand how they've been doing this version switching - using an AppEngine feature called 'traffic splitting'. More details in the comments on #552