Request to add features

mananshr commented 10 years ago

We've been using and reading your googleplay java api, and it works perfectly well in downloading an app from the play store. We were trying to go a step further and see if those downloads could also lead to an increase in install numbers on the play store. But that didn't happen, and in order to make that happen:

We observed the following requests when Gplay market downloads an app:

Apart from these, there was another request, the following, which was being sent at random times. This contained a Base64 encoded string which, as far as we could apprehend had Authtoken, logging id(constant for a device), device_id(gsf_id for the device), and a string that had list of package names installed which we think are protobuffed and we couldn't figure out.

https://android.clients.google.com/market/api/ApiRequest

Here is the trace for one such request :

 POST https://android.clients.google.com/market/api/ApiRequest HTTP/1.1

 Content-Type: application/x-www-form-urlencoded; charset=UTF-8

 User-Agent: Android-Market/2

 X-Public-Android-Id: 6532b095d0193d17

 Content-Length: 7183

 Host: android.clients.google.com

 Connection: Keep-Alive

version=2&request=CuMCCosCRFFBQUFMb0FBQURjQnUxX1JmM3pfYzhhMnpLbmhMamh3UmVpVUxiUDl1YTNheEEwalVyWTBwSW9ieTZNTFNmdlc1MUNnTkhReGN2cTBVek1tZkI0M1R5UnFCSWtwLXZ4UDhRYkFnLTVXTFhYZTVDS0I4U2xRa1Q2QXlPVFQyQlpaTk4zRWJlZTB6SGdlODlYX1JqamtSWkRJMklROFJMSEdQTi10WmNwMmhGZGZVWVY5Wkl2UzIzYjA4RHR4Z09PWXdvZkFCTjJRYU53UW9oN2NsdFlVQkVuTktLMlVLT0RTTGVVZ3E4V2kxYXBnZEFlTzIydElubmgyRGNWRldpaGVGVXlCZG94cnFnEAEYlbuhJiIQMzdhYWVjZDk0YmExNWVjNSoLdmJveDg2dHA6MTYyAmVuOgJVU0IASgBSAFoAYgphbS11bmtub3duahEtNjg2ZTVjZWMyZmZjMjM3MhNKkicTGiR2Mjpjb20ubGlvbWlpLndhdGVyLnB1enpsZS5sb2dpYzoxOjMgAiizvp3Spyg6HWNvbS5saW9taWkud2F0ZXIucHV6emxlLmxvZ2ljQAMUExoddjI6Y29tLnRlc3RhcHAuaGFzb2ZmZXJzNjoxOjMgAiizvp3Spyg6FmNvbS50ZXN0YXBwLmhhc29mZmVyczZAAxQTGh52MjptZS5zb3VuZHdhdmUuc291bmR3YXZlOjE6OTAgAiizvp3Spyg6Fm1lLnNvdW5kd2F2ZS5zb3VuZHdhdmVAWhRTWgdhbmRyb2lkYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWhRjb20uYW5kcm9WTS52bWNvbmZpZ2ABahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oZY29tLmFuZHJvaWQuYmFja3VwY29uZmlybWAQahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oVY29tLmFuZHJvaWQuYmx1ZXRvb3RoYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWhNjb20uYW5kcm9pZC5icm93c2VyYBBqG1llMDNmb1hUaHFqZjdtdUdTOWhiQ19xbHI0RVRTWhdjb20uYW5kcm9pZC5jYWxjdWxhdG9yMmAQahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oUY29tLmFuZHJvaWQuY2FsZW5kYXJgEGobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaEmNvbS5hbmRyb2lkLmNhbWVyYWABahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oZY29tLmFuZHJvaWQuY2VydGluc3RhbGxlcmAQahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oUY29tLmFuZHJvaWQuY29udGFjdHNgEGobV3phTV95MmlhR21XdkpYcXdaRHFwUFZqRC1VVFNaGGNvbS5hbmRyb2lkLmRlZmNvbnRhaW5lcmAQahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oVY29tLmFuZHJvaWQuZGVza2Nsb2NrYMsBahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oRY29tLmFuZHJvaWQuZW1haWxgkIMZahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oUY29tLmFuZHJvaWQuZXhjaGFuZ2VgoMIeahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oTY29tLmFuZHJvaWQuZ2FsYXh5NGABahtXemFNX3kyaWFHbVd2Slhxd1pEcXBQVmpELVVUU1oTY29tLmFuZHJvaWQuZ2FsbGVyeWAQaht0NTMwcUM2UXRYNm5aU1dyY0Rlckk0cEM5ZE1UU1oWY29tLmFuZHJvaWQuaHRtbHZpZXdlcmAQahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oYY29tLmFuZHJvaWQuaW5wdXRkZXZpY2VzYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWh1jb20uYW5kcm9pZC5pbnB1dG1ldGhvZC5sYXRpbmAQahtXemFNX3kyaWFHbVd2Slhxd1pEcXBQVmpELVVUU1oeY29tLmFuZHJvaWQuaW5wdXRtZXRob2QucGlueWluYBBqG1d6YU1feTJpYUdtV3ZKWHF3WkRxcFBWakQtVVRTWhRjb20uYW5kcm9pZC5rZXljaGFpbmAQahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oUY29tLmFuZHJvaWQubGF1bmNoZXJgEGobV3phTV95MmlhR21XdkpYcXdaRHFwUFZqRC1VVFNaJGNvbS5hbmRyb2lkLmxpdmV3YWxscGFwZXIubWljcm9iZXNnbGAPahtuRUZ5bEhHeWtQeUxJQXZQWXRLVWQtc1lHdWtUU1oWY29tLmFuZHJvaWQubWFnaWNzbW9rZWAQahtXemFNX3kyaWFHbVd2Slhxd1pEcXBQVmpELVVUU1oPY29tLmFuZHJvaWQubW1zYBBqG1llMDNmb1hUaHFqZjdtdUdTOWhiQ19xbHI0RVRTWhFjb20uYW5kcm9pZC5tdXNpY2AQahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oTY29tLmFuZHJvaWQubXVzaWNmeGCgUWobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaFGNvbS5hbmRyb2lkLm11c2ljdmlzYBBqG1d6YU1feTJpYUdtV3ZKWHF3WkRxcFBWakQtVVRTWhZjb20uYW5kcm9pZC5ub2lzZWZpZWxkYAFqG1d6YU1feTJpYUdtV3ZKWHF3WkRxcFBWakQtVVRTWhxjb20uYW5kcm9pZC5wYWNrYWdlaW5zdGFsbGVyYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWhVjb20uYW5kcm9pZC5waGFzZWJlYW1gAWobV3phTV95MmlhR21XdkpYcXdaRHFwUFZqRC1VVFNaEWNvbS5hbmRyb2lkLnBob25lYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWiJjb20uYW5kcm9pZC5wcm92aWRlcnMuYXBwbGljYXRpb25zYBBqG1d6YU1feTJpYUdtV3ZKWHF3WkRxcFBWakQtVVRTWh5jb20uYW5kcm9pZC5wcm92aWRlcnMuY2FsZW5kYXJgEGobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaHmNvbS5hbmRyb2lkLnByb3ZpZGVycy5jb250YWN0c2AQahtXemFNX3kyaWFHbVd2Slhxd1pEcXBQVmpELVVUU1ofY29tLmFuZHJvaWQucHJvdmlkZXJzLmRvd25sb2Fkc2AQaht0NTMwcUM2UXRYNm5aU1dyY0Rlckk0cEM5ZE1UU1oiY29tLmFuZHJvaWQucHJvdmlkZXJzLmRvd25sb2Fkcy51aWAQaht0NTMwcUM2UXRYNm5aU1dyY0Rlckk0cEM5ZE1UU1oZY29tLmFuZHJvaWQucHJvdmlkZXJzLmRybWAQaht0NTMwcUM2UXRYNm5aU1dyY0Rlckk0cEM5ZE1UU1obY29tLmFuZHJvaWQucHJvdmlkZXJzLm1lZGlhYP0Daht0NTMwcUM2UXRYNm5aU1dyY0Rlckk0cEM5ZE1UU1oeY29tLmFuZHJvaWQucHJvdmlkZXJzLnNldHRpbmdzYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWh9jb20uYW5kcm9pZC5wcm92aWRlcnMudGVsZXBob255YBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWiRjb20uYW5kcm9pZC5wcm92aWRlcnMudXNlcmRpY3Rpb25hcnlgEGobV3phTV95MmlhR21XdkpYcXdaRHFwUFZqRC1VVFNaFWNvbS5hbmRyb2lkLnByb3Zpc2lvbmAQahtKeGx1T0d1SFhuYXQ5d0RuNm9Ua3h1N2pQZm9UU1oUY29tLmFuZHJvaWQuc2V0dGluZ3NgEGobSnhsdU9HdUhYbmF0OXdEbjZvVGt4dTdqUGZvVFNaH2NvbS5hbmRyb2lkLnNoYXJlZHN0b3JhZ2ViYWNrdXBgEGobSnhsdU9HdUhYbmF0OXdEbjZvVGt4dTdqUGZvVFNaE2NvbS5hbmRyb2lkLnNtc3B1c2hgEGobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaGWNvbS5hbmRyb2lkLnNvdW5kcmVjb3JkZXJgEGobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaFGNvbS5hbmRyb2lkLnN5c3RlbXVpYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWhNjb20uYW5kcm9pZC52ZW5kaW5nYJW7oSZqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWhdjb20uYW5kcm9pZC52aWRlb2VkaXRvcmALahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUU1oXY29tLmFuZHJvaWQudm9pY2VkaWFsZXJgEGobWWUwM2ZvWFRocWpmN211R1M5aGJDX3FscjRFVFNaFmNvbS5hbmRyb2lkLnZwbmRpYWxvZ3NgEGobSnhsdU9HdUhYbmF0OXdEbjZvVGt4dTdqUGZvVFNaFWNvbS5hbmRyb2lkLndhbGxwYXBlcmAQahtXemFNX3kyaWFHbVd2Slhxd1pEcXBQVmpELVVUU1ogY29tLmFuZHJvaWQud2FsbHBhcGVyLmhvbG9zcGlyYWxgEGobV3phTV95MmlhR21XdkpYcXdaRHFwUFZqRC1VVFNaIGNvbS5hbmRyb2lkLndhbGxwYXBlci5saXZlcGlja2VyYBBqG0p4bHVPR3VIWG5hdDl3RG42b1RreHU3alBmb1RTWhtjb20uY3lhbm9nZW5tb2QuZmlsZW1hbmFnZXJgZWobSnhsdU9HdUhYbmF0OXdEbjZvVGt4dTdqUGZvVFNaHWNvbS5leGFtcGxlLmFuZHJvaWQubGl2ZWN1YmVzYBBqG1llMDNmb1hUaHFqZjdtdUdTOWhiQ19xbHI0RVRTWiljb20uZ29vZ2xlLmFuZHJvaWQuYXBwcy5nZW5pZS5nZW5pZXdpZGdldGCfCmobSkxza3dGNUg0Szc2YUtXS2RtRjUyYllUcGdBVFNaIGNvbS5nb29nbGUuYW5kcm9pZC5hcHBzLnVwbG9hZGVyYMC4AmobT0pHS1JUMEhHWk5VLUxHYThGN0dWaXp0VjRnVFNaGWNvbS5nb29nbGUuYW5kcm9pZC5iYWNrdXBgEGobSnhsdU9HdUhYbmF0OXdEbjZvVGt4dTdqUGZvVFNaG2NvbS5nb29nbGUuYW5kcm9pZC5mZWVkYmFja2AQahtPSkdLUlQwSEdaTlUtTEdhOEY3R1ZpenRWNGdUU1onY29tLmdvb2dsZS5hbmRyb2lkLmdvb2dsZXF1aWNrc2VhcmNoYm94YIC5x19qG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWhZjb20uZ29vZ2xlLmFuZHJvaWQuZ3NmYBBqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWhxjb20uZ29vZ2xlLmFuZHJvaWQuZ3NmLmxvZ2luYBBqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWhtjb20uZ29vZ2xlLmFuZHJvaWQubG9jYXRpb25g1ghqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWiJjb20uZ29vZ2xlLmFuZHJvaWQubWFydmluLnRhbGtiYWNrYERqG20wSk1MU2V0VWFRcU0zNEx0cGtjZHV5a1JHRVRTWiVjb20uZ29vZ2xlLmFuZHJvaWQub25ldGltZWluaXRpYWxpemVyYBBqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWh9jb20uZ29vZ2xlLmFuZHJvaWQucGFydG5lcnNldHVwYBBqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWh5jb20uZ29vZ2xlLmFuZHJvaWQuc2V0dXB3aXphcmRgggFqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWiljb20uZ29vZ2xlLmFuZHJvaWQuc3luY2FkYXB0ZXJzLmJvb2ttYXJrc2AQahtPSkdLUlQwSEdaTlUtTEdhOEY3R1ZpenRWNGdUU1ooY29tLmdvb2dsZS5hbmRyb2lkLnN5bmNhZGFwdGVycy5jYWxlbmRhcmAPahtPSkdLUlQwSEdaTlUtTEdhOEY3R1ZpenRWNGdUU1ooY29tLmdvb2dsZS5hbmRyb2lkLnN5bmNhZGFwdGVycy5jb250YWN0c2AQahtPSkdLUlQwSEdaTlUtTEdhOEY3R1ZpenRWNGdUU1oXY29tLmdvb2dsZS5hbmRyb2lkLnRhbGtgygJqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWhZjb20uZ29vZ2xlLmFuZHJvaWQudHRzYBBqG09KR0tSVDBIR1pOVS1MR2E4RjdHVml6dFY0Z1RTWh5jb20uZ29vZ2xlLmFuZHJvaWQudm9pY2VzZWFyY2hggJL0AWobT0pHS1JUMEhHWk5VLUxHYThGN0dWaXp0VjRnVFNaF2NvbS5ub3NodWZvdS5hbmRyb2lkLnN1YC9qG1h4RV93c0lLZkp2WktCa2lhaktoa0V0MTc0c1RTWg1jb20uc3ZveC5waWNvYAFqG1llMDNmb1hUaHFqZjdtdUdTOWhiQ19xbHI0RVRTWhRjb20udGYudGhpbmtkcm9pZC5zZ2DnswpqGzl2UVlPS0QyS3ozMTViZVJfXy1iYlZ4OG1aY1RTWhdqcC5jby5vbXJvbnNvZnQub3BlbndubmAQahtZZTAzZm9YVGhxamY3bXVHUzloYkNfcWxyNEVUcAQU&

Your code simulates the following 5 requests:

After using your functions for the above requests we tried to simulate the following requests ourselves:

-https://android.clients.google.com/fdfe/log

   This request contains a protobuffed string(as per our understanding) which seems to be some kind of a confirmation for downloading the app to the device.

   This is how the string looks: = ( 4confirmFreeDownload?doc=com.google.android.apps.maps

-https://android.clients.google.com/market/api/ApiRequest

   We tried to construct the Base64 encoded string, and the protobuffed string for apps that are installed on a particular device, but recieved a 400 response 

   We replicated a request sent by a Device without changes but we got 400 response.

   (We could not figure out how to get the logging_id as well.)

We need help in reconstructing the same request, and understanding how the the data being sent in the request is formed so that we can figure out the last three requests and complete the install cycle.

Any help would be highly appreciated since you're the ultimate authority on this. :-)

Akdeniz commented 10 years ago

Hi there,

First I want to apologize that I havent been able to look what is going on issues of this project for SO long because of my daily job and other things :/

Anyway here is my findings about your issue:

https://android.clients.google.com/market/api/ApiRequest

Android sends information about installed applications within base64 encoded request to this url. To be precise; request field is base64 encoded RequestProto structure as you can observe below.

RequestPropertiesProto contains information about device, to be used for some kind of authentication i think. Here "logging_id" is hexedecimal of random long number which is provided while device is checked in! (I ignored this value in my code because of not caring about logging until now, but it can easily be added.)

This call request is being made at random times and according to response structure (ResponseProto>ContentSyncResponseProto), it only contains update information about each applications. So I dont think this call is used for updating download count at server side. But I may be wrong about this, so further testing is required.

https://android.clients.google.com/fdfe/log

This url is called at every download operation and likely to be used for download count. It contains LogRequest structure.

I tried to generate some calls to these urls to see if download count of my purposely published application on google play is changed.

I had a diffuculty on authenticating ApiRequest url which always returns me HTTP-403 error. I need to find out what is wrong there..

I successfully generated a request to second url but Google updates statistics about an application at daily basis. So I have to wait until tomorrow to see if it has any affect on download count.

Proto structures:

message ContentSyncRequestProto {
  optional bool incremental = 1;
  repeated group AssetInstallState = 2 {
    optional string assetId = 3;
    optional int32 assetState = 4;
    optional int64 installTime = 5;
    optional int64 uninstallTime = 6;
    optional string packageName = 7;
    optional int32 versionCode = 8;
    optional string assetReferrer = 9;
  }
  repeated group SystemApp = 10 {
    optional string packageName = 11;
    optional int32 versionCode = 12;
    repeated string certificateHash = 13;
  }
  optional int32 sideloadedAppCount = 14;
}
message ContentSyncResponseProto {
  optional int32 numUpdatesAvailable = 1;
}

message RequestPropertiesProto {
  optional string userAuthToken = 1;
  optional bool userAuthTokenSecure = 2;
  optional int32 softwareVersion = 3;
  optional string aid = 4;
  optional string productNameAndVersion = 5;
  optional string userLanguage = 6;
  optional string userCountry = 7;
  optional string operatorName = 8;
  optional string simOperatorName = 9;
  optional string operatorNumericName = 10;
  optional string simOperatorNumericName = 11;
  optional string clientId = 12;
  optional string loggingId = 13;
}
message RequestProto {
  optional RequestPropertiesProto requestProperties = 1;
  repeated group Request = 2 {
    optional ContentSyncRequestProto contentSyncRequest = 9;
  }
}

message ResponsePropertiesProto {
  optional int32 result = 1;
  optional int32 maxAge = 2;
  optional string etag = 3;
  optional int32 serverVersion = 4;
  optional int32 maxAgeConsumable = 6;
  optional string errorMessage = 7;
  repeated InputValidationError errorInputField = 8;
}
message ResponseProto {
  repeated group Response = 1 {
    optional ResponsePropertiesProto responseProperties = 2;
    optional ContentSyncResponseProto contentSyncResponse = 8;
  }
}
//////////////////////////////////////////
message ClickLogEvent {
  optional int64 eventTime = 1;
  optional string url = 2;
  optional string listId = 3;
  optional string referrerUrl = 4;
  optional string referrerListId = 5;
}
message LogRequest {
  repeated ClickLogEvent clickEvent = 1;
}

NaomiLi commented 9 years ago

Sorry to ask for help !!! im confused with how to make a "ContentSyncRequestProto" request, always returns 403, forbidden. what's the rule of SystemApp .certificateHash . but i replicated a ContentSyncRequestProto request sent by a Device, it returns 200. thanks.

ouyangzhanzhu commented 8 years ago

Sorry to ask for help !!! How to successfully call this interface (https://android.clients.google.com/fdfe/log), what parameters can be passed to share the solution? thanks. @Akdeniz

azaleski9 commented 8 years ago

That's quite easy

private static final String FDFE_URL = "https://android.clients.google.com/fdfe/";
public static final String LOG_URI = FDFE_URL + "log";
`public Log.LogResponse log(Log.LogRequest log) {
        try {
        ResponseWrapper responseWrapper = executePOSTRequest(LOG_URI,
                log.toByteArray(), "application/x-protobuf");
            return responseWrapper.getPayload().getLogResponse();
        } catch (Exception ex) {

        }
        return null;
    }

public static Log.ClickLogEvent getConfirmFreeDownloadEvent(String app) {
        return Log.ClickLogEvent.newBuilder().setEventTime(System.currentTimeMillis()).setUrl("confirmFreeDownload?doc=" + app).build();
    }

//Can be added in CheckinWorker for instance
    Log.LogRequest log = Log.LogRequest.newBuilder().addClickEvent(getConfirmFreeDownloadEvent(pkg)).build();
            api.Log(log); //api is your GooglePlayAPI

`

However, does anybody know how to make ContentSyncRequest, because it gives me always error 403 code. That's my protobuf code:

@Akdeniz @ouyangzhanzhu

requestProperties {
  userAuthToken: "lwMPdRVONlT7b0IZp5ujsoUTPY_fEKClTAjuoWM6A5J3ldX8bWIiZcCinOxyg92aPjJlbw."
  userAuthTokenSecure: false
  softwareVersion: 2009011
  aid: "32F7967C1BDF09DD"
  productNameAndVersion: "shamu:22"
  userLanguage: "en"
  userCountry: "US"
  operatorName: "Plus"
  simOperatorName: "Plus"
  operatorNumericName: "26001"
  simOperatorNumericName: "26001"
  clientId: "am-android-google"
loggingId: "" //random here
}
request {
  contentSyncRequest {
    AssetInstallState {
      assetId: "v2:com.google.android.gms:1:9452230"
      installTime: 2016073043
      packageName: "com.google.android.gms"
    }
    sideloadedAppCount: 0
  }
}`

Akdeniz / google-play-crawler