search

JodelRaccoons / jodel_api

Unoffical Python Interface to the Jodel API
https://github.com/nborrmann/jodel_api
MIT License
18 stars 7 forks source link

Error when creating new account or refreshing tokens #14

Closed ttiasg closed 2 years ago

ttiasg commented 3 years ago 
  File "jodel_api.py", line 59, in __init__
    r = self.refresh_all_tokens(pushtoken, **kwargs)
  File "jodel_api.py", line 174, in refresh_all_tokens
    raise Exception(resp)
Exception: (401, {'error': 'Firebase UID not valid', 'metadata': None})

Tested with secret from Version 7.7.2

Morgulisan commented 3 years ago

I was afraid it would be just me. Same error occures on the Node.js version of this api.

Unbrick commented 3 years ago

I can replicate that issue using the key swbBCdBLdtvSqgflkjyrvVwiVHMZSQDQzQWsPiMg from version 7.9.2 . It also seems like the hmac key did not change since the version updates to 7.x.x .

I'll try to look into this but afaik Jodel tries to switch to a email based login procedure. So the registration endpoint currently used might be deprecated and therefore not working anymore.

ToasteR1032 commented 3 years ago

I took a look into this. Jodel indeed switched to a email based login, where you have to enter your email address and click on a confirmation link that was sent to this address.

I collected the relevant requests (Jodel version 7.1.1):

After entering email:

POST https://www.googleapis.com/identitytoolkit/v3/relyingparty/getOobConfirmationCode?key=CIdfHyHLBp02AFFC8fVUxyRqG7vDZ_ED_vLbeF0

Request:

{
    "androidInstallApp": true,
    "androidMinimumVersion": "5.116.0",
    "androidPackageName": "com.tellm.android.app",
    "canHandleCodeInApp": true,
    "continueUrl": "https://jodel.com/app/magic-link-fallback",
    "email": "<email that I entered>",
    "requestType": 6
}

Response:

{
    "email": "<email that I entered>",
    "kind": "identitytoolkit#GetOobConfirmationCodeResponse"
}

The sent link to the given email address looks like the following: https://ae3ts.app.goo.gl/?link=https://tellm-android.firebaseapp.com/__/auth/action?apiKey%3DAIzaSyBC5AfciIsT15NSwrfhLhsLG5UtFisbeSA%26mode%3DsignIn%26oobCode%3DamUkHRnDI-zfe4O6OrCwf1nLVW4aCzJy9G4_kBPaAfkCCCF7c_ItGA%26continueUrl%3Dhttps://jodel.com/app/magic-link-fallback%26lang%3Den&apn=com.tellm.android.app&amv=5.116.0

Confirming email:

POST https://www.googleapis.com/identitytoolkit/v3/relyingparty/emailLinkSignin?key=CIdfHyHLBp02AFFC8fVUxyRqG7vDZ_ED_vLbeF0

(Same key as in request before)

Request:

{
    "email": "<email that I entered>",
    "oobCode": "amUkHRnDI-zfe4O6OrCwf1nLVW4aCzJy9G4_kBPaAfkCCCF7c_ItGA"
}

(obbCode is the one from the confirmation link)

Response:

{
    "email": "<email that I entered>",
    "expiresIn": "3600",
    "idToken": "<some JWT>",
    "isNewUser": false,
    "kind": "identitytoolkit#EmailLinkSigninResponse",
    "localId": "5Necxt3cK3ZicKNJ05oLyvZfulL2",
    "refreshToken": "BCzBnAgI821FcpKMFojgZil4wjiAzp830TRfohWP6Npu64y_kg-2jbRaL7jYgURrGOyu7_6MY8lkf_dtCV2zakJFbzC7Ro6ZHw76lpwXji0l_SA3nGPmb03L30_NVXNtWYK93CDK7h7_hxTa3JS87uuXmdZ9C3I0-U2awAzfamrV1RBJtu2o3swBLyQq8f6AHnQODe2f9hRFB2yXFvjFXj7bRpSq1scuua1WB2sY7_XJHZe6C4HC1ks"
}

Note that isNewUser is probably false because I already entered this email before.

Getting account information:

POST https://www.googleapis.com/identitytoolkit/v3/relyingparty/getAccountInfo?key=CIdfHyHLBp02AFFC8fVUxyRqG7vDZ_ED_vLbeF0

Request:

{
    "idToken": "<same idToken as before>"
}

Response:

{
    "kind": "identitytoolkit#GetAccountInfoResponse",
    "users": [
        {
            "createdAt": "1629736451758",
            "email": "<email that I entered>",
            "emailLinkSignin": true,
            "emailVerified": true,
            "lastLoginAt": "1629737874742",
            "lastRefreshAt": "2021-08-23T16:57:54.742Z",
            "localId": "5Necxt3cK3ZicKNJ05oLyvZfulL2",
            "providerUserInfo": [
                {
                    "email": "<email that I entered>",
                    "federatedId": "<email that I entered>",
                    "providerId": "password",
                    "rawId": "<email that I entered>"
                }
            ],
            "validSince": "1629736451"
        }
    ]
}

Refreshing token:

POST https://securetoken.googleapis.com/v1/token?key=AIzaSyALHp02FYFC7gVUxyRqG8vDZ_FF_vLaiF0

Request:

{
    "grantType": "refresh_token",
    "refreshToken": "BCzBnAgI821FcpKMFojgZil4wjiAzp830TRfohWP6Npu64y_kg-2jbRaL7jYgURrGOyu7_6MY8lkf_dtCV2zakJFbzC7Ro6ZHw76lpwXji0l_SA3nGPmb03L30_NVXNtWYK93CDK7h7_hxTa3JS87uuXmdZ9C3I0-U2awAzfamrV1RBJtu2o3swBLyQq8f6AHnQODe2f9hRFB2yXFvjFXj7bRpSq1scuua1WB2sY7_XJHZe6C4HC1ks"
}

Response:

{
    "access_token": "<some JWT>",
    "expires_in": "3600",
    "id_token": "<some JWT, same as access_token>",
    "project_id": "425112442765",
    "refresh_token": "ACzBnCgI732EcpFNKojgZil4wjiAzp830TRfohWP6Npu64y_kg-2jbRaL7jYgURrGOyu7_6MY8lkf_dtCV2zakJFbzD7Ro6ZHw76lpwXji0l_SA3nGPmb03L30_MVYNtWnK93DDK7h7_hxTa3JS87uuXmdZ9C3I0-U2awCzrmarV1TCJtu2o3swBLyQq8f6AHnQODe2f9hRFB2yXFvjFXj7bRpSq1scuua1OW2sY7_XJHZe6C4HC1ks",
    "token_type": "Bearer",
    "user_id": "5Necxt3cK3ZicKNJ05oLyvZfulL2"
}

"Creating" account:

POST: https://api.jodelapis.com/api/v2/users/

Request:

{
    "adId": "810dec43-c6f3-4c59-a4e2-636a0c67f351",
    "client_id": "31d7a68e-1e02-4c17-9ba0-6a3125261b26",
    "device_uid": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "firebaseJWT": "<same as previous access_token/id_token>",
    "firebase_uid": "5Necxt3cK3ZicKNJ05oLyvZfulL2",
    "iid": "cV6yKJJPSCCv1Cg_n_m6cX:BEA91bHFpYybIJzB9cS4QO6M4yrCIDuK3k9C3hYmhvL8NZxLDx0uYcFDCZPUZd7KA4LpjdikGCTG0zSg79dCYT8FOmA24mHXL8ETD0QaLpZOXowOpDpjArooADYJ7X4a7P3g8x4f6x2A",
    "iid_provider": "google",
    "language": "en-US",
    "location": {
        "city": "Wien",
        "country": "AT",
        "loc_accuracy": 5.0,
        "loc_coordinates": {
            "lat": 48.2081733,
            "lng": 16.3738183
        }
    },
    "registration_data": {
        "campaign": "",
        "channel": "",
        "feature": "",
        "provider": "branch.io",
        "referrer_branch_id": "",
        "referrer_id": ""
    },
    "registration_type": "signup"
}

Can't say anything about the response here because I was using an outdated Jodel version (7.1.1) with an HMAC incompatible to the API (so I got the "Signed request expected" error) but this endpoint is working just as before.

The only difference is in the request. We have two additional fields here - firebaseJWT and firebase_uid (= localId).

I tried adding those fields to the payload inside the refresh_all_tokens function.

Turned out that it doesn't work with the firebaseJWT. However, it does work if we just include the firebase_uid. We can already get this firebase_uid with the request that confirms the email. In this response this uid is called localId.

To sum up, it seems that we need two additional requests to get the creation of users working again:

The retrieved localId can then be used as the firebase_uid inside the payload for the /v2/users/ endpoint.

springjools commented 3 years ago

Thanks for the write up. I just wanted to comment that I appreciate that somebody else works with this. It's likely a cat and mouse game, so I found out that Jodel Gmbh has 38 employees.

Unbrick commented 3 years ago

@ToasteR1032 I'm currently reversing the Android-App, maybe there is another possibility of registration without performing the out-of-bands procedure in place. Otherwise the only option would be the implementation of some sort of IMAP client which automatically fetches the verification mail. Anyway, great RE work!

Unbrick commented 3 years ago

Maybe a quick update on this topic: I reverse engineered parts of the iOS app and got to the following conclusions:

The iOS app is using a technique called Apple Device Check which can be used to idendify and verify the validity of a device. In this process the app acquires a apple_device_token which is encrypted by Apple. This token is required to create new accounts and is sent within the initial account creation request to the server. The server then verifys this token against the Apple server. 

POST /api/v2/users HTTP/1.1
Host: api.jodelapis.com
Content-Type: application/json
X-Timestamp: 2021-11-16T19:32:36Z
Accept: */*
X-Location: xxx;xxx
X-Authorization: HMAC BB88355AF799E6929C1CCDA4012E7AE7CAF8D4B1
Accept-Encoding: gzip, deflate
Accept-Language: de-DE;q=1
X-Api-Version: 0.2
X-Client-Type: ios_7.26
Content-Length: 3427
User-Agent: Jodel/7.26 (iPhone; iOS 14.6; Scale/2.00)
Connection: close

{
    "location": {
        "loc_coordinates": {
            "lat":0,
            "lng":0
        },
        "loc_accuracy": 65,
        "country": "DE"
    },
    "adId": "00000000-0000-0000-0000-000000000000",
    "apple_device_token": "Ag[...]==",
    "device_uid": "DEaUZEhrUGSlTGxjGJ3Qs0ceRRBfDNBe674M0ShnLgM=",
    "language": "de-DE",
    "client_id": "cd871f92-a23f-4afc-8fff-51ff9dc9184e"
}

The device_uid is pretty simple, within the app its generated using 

SHA_256('45C2B096-DFDA-4B1A-A404-26FB4328F358' + (new UUID().replace('-','')) + '0984FC52-FF93-44EB-BD44-3A5C521BAC5E')

Afaik spoofing the apple_device_token is not possible without using a real device. But as stated in the developer docs, not all devices do support Apple Device Check and therefore the developer should implement a check prior using it. Hooking some functions of the binary, modifying the DCDevice.isSupported() method to always return false resulted in the following VALID registration request:

POST /api/v2/users HTTP/1.1
Host: api.jodelapis.com
Content-Type: application/json
X-Timestamp: 2021-11-16T14:12:50Z
Accept: */*
X-Location: 0;0
X-Authorization: HMAC 904E4B6FF19DA76E6759E3C4C29A259D5DFBD8EC
Accept-Encoding: gzip, deflate
Accept-Language: de-DE;q=1
X-Api-Version: 0.2
X-Client-Type: ios_7.26
Content-Length: 293
User-Agent: Jodel/7.26 (iPhone; iOS 14.6; Scale/2.00)
Connection: close

{
    "location": {
        "loc_coordinates": {
            "lat": 0,
            "lng": 0
        },
        "loc_accuracy": 65,
        "country": "DE"
    },
    "adId": "00000000-0000-0000-0000-000000000000",
    "device_uid": "HkpPAf+kVOJFNI485ZD9MppVm5cEhqTPyfhuhIeh37A=",
    "language": "de-DE",
    "client_id": "cd871f92-a23f-4afc-8fff-51ff9dc9184e"
}

As device check is not supported on the now used version, the apple_device_token is not included anymore within the request. But trying to reissue this request with a changed device_uid, the server responds with a "Signed request expected" message. While reissuing the same exact request i used the same HMAC key used in the iOS app and the same version string.

Not sure what is wrong with my signing, might also have to do with my device already beeing registered. Maybe the JodelApp is pulling parts of the UUID out of the iCloud. If i can create a new initial registration this API should be working again.

springjools commented 3 years ago

I assume the device token can be used to identify a single mobile phone, and therefore in many cases a person. Is it compliant with GDPR to maintain a database with device tokens? I know this is out of scope for this purpose, but there could also be a legal route to pursue to try to make companies not use these kind of identification methods.

Unbrick commented 3 years ago

Regarding GDPR, as far as i know only apple can decrypt them and associate them with devices. Apple stores two value bits server sided and the Jodel server can query whether a specific device has a value bit set on the Apple server using the device token. Currently i'm not sure how or if the apple device token is associated with the Jodel account but within the API requests no personal data is transmitted.

Maybe the used tracking (afaik branch.io) might violate GDPR policies as they track a shitload of data (device data, IP address in wifi and much more) but I'm not deep enough in the topic to know whether this is any kind of violation. And it also doesn't affect the registration process, therefore it wont help with fixing this bug 😁

ttiasg commented 2 years ago

Thanks to all for looking into this.

What I found out is, that it is indeed possible to create a new account with the extracted secret from the iOS App when using the iOS API Wrapper. https://github.com/marbink/jodel_ios_api

I'm not entirely sure if the signing routine is somewhat different to our version here and why it wasn't working for @Unbrick

Anyway, bad news is that after creating the account, it seems that one gets shadow banned immediately. Creating a post results in a valid response, but get_user_config()[1]["user_blocked"] returns True and the post never shows up.

What I also noticed is, that after creating a new Jodel Account on my iDevice while running mitm + a certificate unpinner, the account also immediately got banned. Might have something to do with certificates as the App is also downloading two certs before the actual call to the /users Endpoint. My knowledge is limited on that matter so thats just speculation.

Unbrick commented 2 years ago

@ttiasg Thanks for this update!

Based on the jodel_ios_api and own requests i now switched the entire registration to iOS based. Vertification of the accounts is still done using the GCM part, seems like the API is fine using this mixture. I committed the changes directly to master, feel free to try it out!

Running the tests gives a quite promising result: 

=========================================================================== test session starts ===========================================================================
platform win32 -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- \jodel_api\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: \jodel_api, configfile: setup.cfg
plugins: flaky-3.7.0
collected 30 items                                                                                                                                                         

test/test_jodel_api.py::TestUnverifiedAccount::test_reinitalize PASSED                                                                                               [  3%]
test/test_jodel_api.py::TestUnverifiedAccount::test_refresh_access_token PASSED                                                                                      [  6%]
test/test_jodel_api.py::TestUnverifiedAccount::test_set_location PASSED                                                                                              [ 10%]
test/test_jodel_api.py::TestUnverifiedAccount::test_read_posts_recent PASSED                                                                                         [ 13%]
test/test_jodel_api.py::TestUnverifiedAccount::test_get_my_posts PASSED                                                                                              [ 16%]
test/test_jodel_api.py::TestUnverifiedAccount::test_popular_after FAILED                                                                                             [ 20%]
test/test_jodel_api.py::TestUnverifiedAccount::test_channel_after FAILED                                                                                             [ 23%]
test/test_jodel_api.py::TestUnverifiedAccount::test_get_posts_channel FAILED                                                                                         [ 26%]
test/test_jodel_api.py::TestUnverifiedAccount::test_get_pictures PASSED                                                                                              [ 30%]
test/test_jodel_api.py::TestUnverifiedAccount::test_get_channels PASSED                                                                                              [ 33%]
test/test_jodel_api.py::TestUnverifiedAccount::test_set_get_config PASSED                                                                                            [ 36%]
test/test_jodel_api.py::TestUnverifiedAccount::test_notifications PASSED                                                                                             [ 40%]
test/test_jodel_api.py::TestUnverifiedAccount::test_post_details PASSED                                                                                              [ 43%]
test/test_jodel_api.py::TestUnverifiedAccount::test_post_details_v3 PASSED                                                                                           [ 46%]
test/test_jodel_api.py::TestUnverifiedAccount::test_share_url PASSED                                                                                                 [ 50%]
test/test_jodel_api.py::TestUnverifiedAccount::test_pin PASSED                                                                                                       [ 53%]
test/test_jodel_api.py::TestUnverifiedAccount::test_switch_notifications PASSED                                                                                      [ 56%]
test/test_jodel_api.py::TestUnverifiedAccount::test_verify FAILED                                                                                                    [ 60%]
test/test_jodel_api.py::TestUnverifiedAccount::test_vote PASSED                                                                                                      [ 63%]
test/test_jodel_api.py::TestUnverifiedAccount::test_post_reply PASSED                                                                                                [ 66%]
test/test_jodel_api.py::TestUnverifiedAccount::test_post_channel_img PASSED                                                                                          [ 70%]
test/test_jodel_api.py::TestUnverifiedAccount::test_bad_gateway_retry PASSED                                                                                         [ 73%]
test/test_jodel_api.py::TestUnverifiedAccount::test_bad_gateway_no_retry PASSED                                                                                      [ 76%]
test/test_jodel_api.py::TestUnverifiedAccount::test_follow_channel PASSED                                                                                            [ 80%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_legacy SKIPPED (requires an account uid as environment variable)                                             [ 83%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_my_pin_after SKIPPED (requires an account uid as environment variable)                                       [ 86%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_my_voted_after SKIPPED (requires an account uid as environment variable)                                     [ 90%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_notifications_read SKIPPED (requires an account uid as environment variable)                                 [ 93%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_post_message SKIPPED (requires an account uid as environment variable)                                       [ 96%]
test/test_jodel_api.py::TestLegacyVerifiedAccount::test_vote SKIPPED (requires an account uid as environment variable)                                               [100%]

Three of the 30 tests failed but this also seems to have to do with how the tests are implemented.

Additionally, observing the responses using BurpSuite shows the newly registered user is currently not blocked:

PUT /api/v2/users/pushToken HTTP/1.1
Host: api.go-tellm.com
User-Agent: Jodel/ (iPhone; iOS 12.5.2; Scale/2.00)
Accept-Encoding: gzip, deflate
Accept: */*
Connection: close
Authorization: Bearer 13556922-a9ab7abb-6d1493f9-24c7-4b07-b3df-3b8a987d7431
X-Client-Type: ios_7.26
X-Api-Version: 0.2
X-Timestamp: 2021-11-27T11:46:52Z
X-Authorization: HMAC C68A8C8EB9A6A5E52740740838D9D9C323143756
Content-Type: application/json; charset=UTF-8
Content-Length: 223

{"client_id": "cd871f92-a23f-4afc-8fff-51ff9dc9184e", "push_token": "eJ3gzt6YhJ4:APA91bHugasz7yKdF_yHQkfR5_vsnEehsZua5HOXAubSYaivU72rkYxmFyO-pmbgjlv4RuslLpdvPB2T7XQfsmP3mOI1s_m0mOP7NarYNgmcKRldgbMjqojZf9H4vNdmm8RS-ERzFK_y"}

======================================================================

HTTP/1.1 204 No Content
Server: nginx/1.13.12
Date: Sat, 27 Nov 2021 11:46:54 GMT
Connection: close
Access-Control-Allow-Origin: *
X-Timestamp: 2021-11-27T11:46:54Z
X-User-Blocked: false
X-Feed-Internationalizable: true
X-Feed-Internationalized: false

Lets hope this solution lasts at least for a while. If you want to extract the HMAC-Key and App-Version from the iOS app, i built a frida script to do so.

ttiasg commented 2 years ago

Thank you @Unbrick

Can confirm, everything's working again :)

Unbrick commented 2 years ago

Thanks for the update, i'll close this for now!