search

BehindTheMath / XVoicePlus

5 stars 3 forks source link

Failing to send messages - unable to parse JSON #1

Open numericOverflow opened 6 years ago

numericOverflow commented 6 years ago

Recently, xvoice++ stopped sending messages, and it appears as the attempt to send the SMS is not getting back a JSON object around line 156 in sendGVMessage() . I was able to pull my android logs and I've attached a screenshot of majority of the error messages.

I think I've seen this in the pyGoogleVoice (unofficial python GVoice API) and I'm wondering if google is sending the plain text error message instructing the user to go to their site and prove their not a robot

Is it possible to get a try/catch block to output the error message to debug? Right now it just dies with an unhandled exception so no visibility to what's actually coming back.

I'm not a android developer, but I'd be more than happy to help you test/debug or provide any logs you need to potentially diagnose. I've got a rooted phone so have some leeway on accessing debug logs, etc.

2018_04_19_09 14 10

BehindTheMath commented 6 years ago

It's been a long time since I looked at this project, but I'll see what I can do.

It might be faster to use a proxy like Fiddler or Charles to intercept the traffic on a computer and inspect it.

numericOverflow commented 6 years ago

Ah, good idea. I pulled this snippet of the conversation between xvoice++ and google:

<------ GET /voice/request/user HTTP/1.1 Host: www.google.com User-Agent: Dalvik/2.1.0 (Linux; U; Android) Accept-Encoding: gzip, deflate Connection: keep-alive Accept: / Authorization: GoogleLogin auth=[[REDACTED]].

------> HTTP/1.1 302 Moved Temporarily Location: https://voice.google.com Content-Type: text/html; charset=UTF-8 Content-Encoding: gzip Date: Thu, 19 Apr 2018 15:07:51 GMT Expires: Thu, 19 Apr 2018 15:07:51 GMT Cache-Control: private, max-age=0 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Server: GSE Alt-Svc: hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35" Transfer-Encoding: chunked

<HTML> <HEAD> <TITLE>Moved Temporarily</TITLE> </HEAD> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> <H1>Moved Temporarily</H1> The document has moved <A HREF="https://voice.google.com"&gt;here&lt;/A&gt;. </BODY> </HTML>

<------ POST /voice/sms/send/ HTTP/1.1 Host: www.google.com User-Agent: Dalvik/2.1.0 (Linux; U; Android) Accept-Encoding: gzip, deflate Connection: keep-alive Accept: / Authorization: GoogleLogin auth=[[REDACTED]]. Content-Type: application/x-www-form-urlencoded; charset=utf-8 Content-Length: 52

phoneNumber=%2B15555555555&sendErrorSms=0&text=TextMessageHere

------> HTTP/1.1 405 HTTP method POST is not supported by this URL Content-Type: text/html; charset=UTF-8 Content-Encoding: gzip Date: Thu, 19 Apr 2018 15:07:54 GMT Expires: Thu, 19 Apr 2018 15:07:54 GMT Cache-Control: private, max-age=0 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Server: GSE Alt-Svc: hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35" Transfer-Encoding: chunked

<HTML> <HEAD> <TITLE>HTTP method POST is not supported by this URL</TITLE> </HEAD> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> <H1>HTTP method POST is not supported by this URL</H1> <H2>Error 405</H2> </BODY> </HTML>

BehindTheMath commented 6 years ago

Based on my testing, it looks like the GV API is checking the user-agent string, and sending back an error message if it detects an Android user-agent.

I'm going to try patch it by sending a blank user-agent. In my testing that appears to work.

Of course, if this change is because Google is trying to block uses like this, we may have to switch to spoofing a desktop user-agent. It's also possible that they can change the API completely.

BehindTheMath commented 6 years ago

Is it possible to get a try/catch block to output the error message to debug? Right now it just dies with an unhandled exception so no visibility to what's actually coming back.

I don't think this is a good idea in general, since the response could potentially contain sensitive information, like authorization codes.

You can always clone the repo, add your own debugging code, and recompile the APK.

BehindTheMath commented 6 years ago

I'm going to try patch it by sending a blank user-agent. In my testing that appears to work.

Apparently a setting a blank user-agent with Ion doesn't actually set it. Instead, I set it to spoof Chrome 65 on Windows 10 x64.

I uploaded a new build here. Please try it and tell me if it works.

numericOverflow commented 6 years ago

I just installed your dev version and sent a couple test messages, which seemed to send ok. I'll give it a better workout tomorrow and let you know how it goes.

BehindTheMath commented 6 years ago

@numericOverflow Can I assume it's working as expected?

numericOverflow commented 6 years ago

@BehindTheMath - It's sort of working. I don't see that original JSON error in the logs any more, but xvoice+ isn't reliably retrieving my messages - some come through, others don't, but all show in the official gvoice app. I'm trying to track down exactly what's going on to reproduce consistently.

About the only thing I can find it that I seem to get this error in my logs:

Conscrypt initialization failed.
java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/android/gms/security/ProviderInstaller;
    at com.koushikdutta.ion.conscrypt.ConscryptMiddleware.initialize(ConscryptMiddleware.java:58)
    at com.koushikdutta.ion.conscrypt.ConscryptMiddleware.initialize(ConscryptMiddleware.java:75)
    at com.koushikdutta.ion.conscrypt.ConscryptMiddleware.getSocket(ConscryptMiddleware.java:103)
    at com.koushikdutta.async.http.AsyncHttpClient.executeAffinity(AsyncHttpClient.java:301)
    at com.koushikdutta.async.http.AsyncHttpClient.access$100(AsyncHttpClient.java:51)
    at com.koushikdutta.async.http.AsyncHttpClient$1.run(AsyncHttpClient.java:190)
    at com.koushikdutta.async.AsyncServer.lockAndRunQueue(AsyncServer.java:740)
    at com.koushikdutta.async.AsyncServer.runLoop(AsyncServer.java:758)
    at com.koushikdutta.async.AsyncServer.run(AsyncServer.java:658)
    at com.koushikdutta.async.AsyncServer.access$800(AsyncServer.java:44)
    at com.koushikdutta.async.AsyncServer$14.run(AsyncServer.java:600)
Caused by: java.lang.ClassNotFoundException: Didn't find class "com.google.android.gms.security.ProviderInstaller" on path: DexPathList[[zip file "/data/app/io.behindthemath.xvoiceplus-1/base.apk"],nativeLibraryDirectories=[/vendor/lib64, /system/lib64]]
    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
    ... 11 more
    Suppressed: java.lang.ClassNotFoundException: Didn't find class "com.google.android.gms.security.ProviderInstaller" on path: DexPathList[[dex file "/data/dalvik-cache/xposed_XResourcesSuperClass.dex", dex file "/data/dalvik-cache/xposed_XTypedArraySuperClass.dex"],nativeLibraryDirectories=[/vendor/lib64, /system/lib64]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:504)
        ... 12 more
        Suppressed: java.lang.ClassNotFoundException: com.google.android.gms.security.ProviderInstaller
            at java.lang.Class.classForName(Native Method)
            at java.lang.BootClassLoader.findClass(ClassLoader.java:781)
            at java.lang.BootClassLoader.loadClass(ClassLoader.java:841)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:504)
            ... 13 more
        Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack available
BehindTheMath commented 6 years ago

@numericOverflow Do you have Google Play Services or GApps installed?

Edit: I see you commented here about this error. Based on @koush's comment here and the comment here, it seems like this error just means Ion isn't using Play Services for SSL, and is falling back to use the system's implementation, so there should be no difference in functionality.

I suppose I could add some try...catch blocks to suppress the error from showing.

numericOverflow commented 6 years ago

I do have Google play services, I'm running an LG G4 from VZW that's rooted, but otherwise mostly stock Verizon Lillipop.

I think you're right, I just can't find any other errors in the logs to explain why xvoice+ isn't working consistently, that was all I could find.

BehindTheMath commented 6 years ago

It's been a while since I personally used XVoice++, but if I remember correctly, I had the same issue and I wasn't able to pin down the cause. The only thing I can suggest is to enable all the sync options.

numericOverflow commented 6 years ago

Hmm... I've already got all the sync options enabled. I guess I'll keep trying to track down something helpful to diagnose the issue.

tb-303 commented 6 years ago

Ya, I have sync issues with the dev build, too :( Still on api 23, no issues prior to this changeup for years

BehindTheMath commented 6 years ago

@tb-303 You're having issues with incoming or outgoing or syncing? It sounds like everyone else is just having issues with incoming.

tb-303 commented 6 years ago

No, just issues with incoming sync.

nathanmerrill commented 6 years ago

I tried the new .apk, and I'm also having issues receiving. To be explicit:

If I send from an SMS app, the recipient gets it. The google voice app doesn't show that it was ever sent. However, if they reply, the google voice app does show the reply, but the SMS app doesn't show any reply.

I'd love to help with this, but I have no idea how to get a local environment setup to test this. The only process I can think of is to install Xposed on the android emulator, and then this app, which seems like an ordeal.

nathanmerrill commented 6 years ago

Ok, so I may have made some progress (or I could be on the wrong track entirely).

I think that the method that XVoicePlus hooks into doesn't exist anymore (com.google.android.apps.voice.backends.gcm::a). I exported the Hangouts app, decompiled it, and as far as I can tell, there's no class by that name. I poked around in the Jar, and I think that the new method to listen to is com.google.android.gms.gcm.GcmReceiver::a, but I'm not confident on that.

I wish I could be more confident in the above, but I'm unable to get debugging or logging working at handleLoadPackage(), which apparently happens super early. If anybody has any suggestions on how to get that working, that would be awesome.

nathanmerrill commented 6 years ago

More progress: I believe that the above methods are being called, but I don't think that logging works at that time.

The fundamental problem is that the request in GoogleVoiceManager::retrieveMessages(). It is also getting a JSON error, which is preventing us from reading messages.

There's also a bug relating to Sync poll timer, making it not work (hookBroadcastPermissionCheck would change the UID of the USER_POLL action (it was just supposed to change the SMS_DELIVER action)). I've fixed that locally, but since syncing doesn't work, I haven't submitted a PR.

nathanmerrill commented 6 years ago

I fixed the user agent on all of the queries, but it appears that the query "https://www.google.com/voice/request/messages" doesn't return text messages anymore. It returns voice messages, but there's no more attribute for texts. I tried capturing traffic, but I'm not sure how to bypass SSL (I've tried Packet Capture, but it simply says "No data")

BehindTheMath commented 6 years ago

@nathanmerrill Thank you for working on debugging this. I'm busy with a few other projects, so currently I don't have much time for this one.

I think that the method that XVoicePlus hooks into doesn't exist anymore (com.google.android.apps.voice.backends.gcm::a). I exported the Hangouts app, decompiled it, and as far as I can tell, there's no class by that name. I poked around in the Jar, and I think that the new method to listen to is com.google.android.gms.gcm.GcmReceiver::a, but I'm not confident on that.

Do you mean Hangouts(com.google.android.talk) or Google Voice(com.google.android.apps.googlevoice)? XVoice never hooked Hangouts; it only use Google Voice. I checked GV v5.9.191668881 last week, and it still appears to have the com.google.android.apps.voice.backends.gcm.GcmListenerService class that XVoicePlus hooks. As you mentioned, there is also com.google.android.gms.gcm.GcmReceiver, but I'm not sure what that class does, and at which point in the message processing sequence it is used.

I wish I could be more confident in the above, but I'm unable to get debugging or logging working at handleLoadPackage(), which apparently happens super early. If anybody has any suggestions on how to get that working, that would be awesome.

handleLoadPackage() is called by Xposed early in the boot sequence, as it initializes the module. As far as I know, there's no way to debug it. Logging should work, however, and as you can see from the current code, there are several logging calls which used to work.

More progress: I believe that the above methods are being called, but I don't think that logging works at that time.

I'm not sure about this, since as I mentioned, logging should work.

The fundamental problem is that the request in GoogleVoiceManager::retrieveMessages(). It is also getting a JSON error, which is preventing us from reading messages.

I believe this is due to the user agent string, as mentioned upthread. This should be fixed in the test build.

There's also a bug relating to Sync poll timer, making it not work (hookBroadcastPermissionCheck would change the UID of the USER_POLL action (it was just supposed to change the SMS_DELIVER action)). I've fixed that locally, but since syncing doesn't work, I haven't submitted a PR.

That's interesting. I guess there should be a check for which action it is. Can you post a snippet of what you changed?

I fixed the user agent on all of the queries, but it appears that the query "https://www.google.com/voice/request/messages" doesn't return text messages anymore. It returns voice messages, but there's no more attribute for texts.

I would have to double-check, but I thought it worked when I tried it last week.

I tried capturing traffic, but I'm not sure how to bypass SSL (I've tried Packet Capture, but it simply says "No data")

You need to use a custom root certificate from a network capture program like Fiddler or Charles so you can decrypt the HTTPS traffic.

nathanmerrill commented 6 years ago

Do you mean Hangouts or Google Voice?

You are totally right: I was looking at hangouts.

I would have to double-check, but I thought it worked when I tried it last week.

If you hit that URL (in your browser, assuming you are logged in), it should auto download the JSON.

I was unable to get Fiddler or Charles to work. I may have done something wrong, but it appears that hangouts is detecting it somehow.

On a side note: I've never been able to use this, as I just recently found it. I have the hangouts app installed, and assumed that that was the Google Voice app. That's clearly not the case :)

BehindTheMath commented 6 years ago

Can you try with the GV app and see what still doesn't work?

nathanmerrill commented 6 years ago

Yes, I definitely can tonight. I don't have my machine with Android Studio setup in front of me :)

nathanmerrill commented 6 years ago

@BehindTheMath I reinstalled the APK above, and incoming SMS still isn't working. It's not an error, either, we simply aren't getting the notification. (There's no log for "Handling intent for action INCOMING_VOICE")

BehindTheMath commented 6 years ago

@BehindTheMath I reinstalled the APK above, and incoming SMS still isn't working. It's not an error, either, we simply aren't getting the notification. (There's no log for "Handling intent for action INCOMING_VOICE")

Right, that's the issue everyone else is having.

It appears to me that either com.google.android.apps.voice.backends.gcm.GcmListenerService.a() isn't being called, or the hook isn't being called, but I don't know why.

Does time-based syncing work for you?

nathanmerrill commented 6 years ago

It works on my patched version (but my patch broke outgoing SMS).

On Wed, May 16, 2018, 2:59 PM BehindTheMath notifications@github.com wrote:

@BehindTheMath https://github.com/BehindTheMath I reinstalled the APK above, and incoming SMS still isn't working. It's not an error, either, we simply aren't getting the notification. (There's no log for "Handling intent for action INCOMING_VOICE")

Right, that's the issue everyone else is having.

It appears to me that either com.google.android.apps.voice.backends.gcm.GcmListenerService.a() isn't being called, or the hook isn't being called, but I don't know why.

Does time-based syncing work for you?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BehindTheMath/XVoicePlus/issues/1#issuecomment-389663706, or mute the thread https://github.com/notifications/unsubscribe-auth/AC0XP3d6I4-taQJISTfeKxiSFyw6U7d7ks5tzJMogaJpZM4Tb1fc .

BehindTheMath commented 6 years ago

It works on my patched version (but my patch broke outgoing SMS).

What did you change?

nathanmerrill commented 6 years ago

In XVoicePlus::hookBroadcastPermissionCheck I checked the third parameter (Intent) and checked what the intent name was. I can get you the code in a hour and a half.

On Wed, May 16, 2018, 3:07 PM BehindTheMath notifications@github.com wrote:

It works on my patched version (but my patch broke outgoing SMS).

What did you change?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BehindTheMath/XVoicePlus/issues/1#issuecomment-389666000, or mute the thread https://github.com/notifications/unsubscribe-auth/AC0XP1svJZZ9gIHxLNcrRsOdE8dPu2oYks5tzJUKgaJpZM4Tb1fc .

nathanmerrill commented 6 years ago

`final int ARGUMENT_INDEX_CALLING_UID = 14; final int ARGUMENT_INDEX_INTENT = 2; int callingUid = (int) param.args[ARGUMENT_INDEX_CALLING_UID]; Intent intent = (Intent) param.args[ARGUMENT_INDEX_INTENT];

// Get our UID int appUid = AndroidAppHelper.currentApplication().getPackageManager() .getApplicationInfo(XVOICE_PLUS_PACKAGE, PackageManager.GET_META_DATA).uid; // If the broadcast is from us if ("android.provider.Telephony.SMS_DELIVER".equals(intent.getAction()) && (boolean) callStaticMethod(UserHandle.class, "isSameApp", callingUid, appUid)) {

Log.d(TAG, "Hooking broadcast permissions: Overriding callingUid "
        + callingUid + " with Process.PHONE_UID (UID 1001)");

// Spoof the broadcast as if it's coming from PHONE_UID, so the system will let it through
param.args[ARGUMENT_INDEX_CALLING_UID] = Process.PHONE_UID;

} `

This is the code so far. I can make an APK if you want.

Anyways, I think the error that requires addressing is: BroadcastQueue is showing the following error: Permission Denial: receiving Intent { act=android.provider.Telephony.SMS_DELIVER flg=0x10 (has extras ) } to com.google.android.talk/com.google.android.apps.hangouts.sms.SmsDeliverReceiver requires android.permission.RECEIVE_SMS due to sender io.behind themath.xvoiceplus (uid 1001)

nathanmerrill commented 6 years ago

Ok, that error was actually pretty trivial to fix. Now dealing with Error accessing registered_accounts from GV SharedPreferences

nathanmerrill commented 6 years ago

Ok, so apparently, the path to the GV preferences changed. The fix is in GCMListenerServiceHook line 104 to final String registeredAccounts = gvSharedPrefs.getString("accounts", null); (change registered_accounts to accounts).

Great news though, there are still more issues. https://www.google.com/voice/sms/send/ is now returning a 500 error for me.