search

PromyLOPh / pandora-apidoc

pandora.com API documentation
https://6xq.net/pandora-apidoc/
Do What The F*ck You Want To Public License
48 stars 19 forks source link

New parameters to https://www.pandora.com/api/v1/auth/login #45

Open jasonful opened 3 years ago

jasonful commented 3 years ago

Up until yesterday, both the pandora.com web site and my own code were using the REST API to login with no problem, sending 4 parameters (existingAuthToken, username, password, keepLoggedIn). But now the web site is sending three more parameters named OZ_TC, OZ_DT, OZ_SG. The contents are encoded or encrypted. Without those 3 parameters, my code is now getting back: {"message":"Invalid username and/or password","errorCode":0,"errorString":"AUTH_INVALID_USERNAME_PASSWORD"}

Anyone know what those parameters are?

hacker1024 commented 3 years ago

I've just ran into this issue as well (stopping my 2+ years of work towards my client in its tracks).

I'm inspecting the source code to see how these value are generated - if you want to, you can use my tool to download the sources (the download button is in the top right): https://toolbox.epimetheus.tk/#source_code

hacker1024 commented 3 years ago

There seem to be no significant Web app source changes between version 1.153.0 (the last version I have before the change) and the latest 1.158.1, suggesting that this was a server-side change that's been long coming.

hacker1024 commented 3 years ago

These new properties seem to be used for bot mitigation, in a new feature referred to as "Detector" internally.

hacker1024 commented 3 years ago

There are references to White Ops.

hacker1024 commented 3 years ago

The White Ops report is embedded into the login request - that's what those values are for. The values seem to be generated by a script called pagespeed.js, which is downloaded from a URL generated with a few properties. Here's one:

https://s.hoplon.pandora.com/static/pandora/4.80.9/pagespeed.js?dt=5222681607985354683000&psv=4.80.9&ci=522268&spa=1&pd=acc&mo=2&di=pandora.com&si=SXMP&ui=1069685625

Unfortunately, the script is obfuscated...

hacker1024 commented 3 years ago

Note the "hoplon" subdomain - a Hoplon is a Greek shield.

hacker1024 commented 3 years ago

Another script is accessed by the script linked above: https://s.hoplon.pandora.com/sri/config.json

hacker1024 commented 3 years ago

This may be the end for unofficial REST API usage; the bot detection seems to be highly reliant on the browser environment, and there aren't even public SDKs available.

Is the JSON API's authToken compatible with the REST API? If so, perhaps the initial login can be done with it, as this bot detection seems to be used only for login.

PromyLOPh commented 3 years ago

@hacker1024 Please edit your initial post instead of posting multiple times in a row.

Looking at https://github.com/PromyLOPh/pianobar/issues/236 this was probably overdue. Pandora never really liked 3rd party developers using their web API’s.

Can I remove the documentation for that API from this repository?

hacker1024 commented 3 years ago

Can I remove the documentation for that API from this repository?

It looks like the JSON API authToken is actually compatible with the REST API! That means existing software can use the JSON API for logging in and no other code needs to be updated. I think it's worth keeping the REST documentation up, as it's still usable in this way.

GodStar88 commented 3 years ago

so how can we login pandora?

hacker1024 commented 3 years ago

@GodStar88 Use the JSON API to obtain the userAuthToken, which can be used as the authToken with the REST API.

GodStar88 commented 3 years ago

thank you for your reply, I tried that, but i can't login pandora

hacker1024 commented 3 years ago

@GodStar88 what's going wrong exactly? Can you share your code?

GodStar88 commented 3 years ago

I tried postman, I can't login

perette commented 3 years ago

There are 3 new fields for authentication. If they are missing, the login fails as if it's invalid credentials.

OZ_DT and OZ_TC are retrieved from a JSON response obtained via GET https://s.hoplon.pandora.com/sri/config.json?dt=5222681607985354683000&psv=4.80.9&ci=522268&spa=1&pd=acc&mo=2&di=pandora.com&si=SXMP&ui=985753384

Where:

Repeatedly using the same Hoplon URL parameters seems to work fine, although I am suspicious this may change as the web client version number changes.

The response from Hoplon is malformed JSON; string substitute "\\x26" -> "&" and "\\x3D" -> "=" to make it valid (These strings are presented here as quoted for C/C++; i.e., these are 4-character strings starting with a single backslash). In the resulting JSON, members ozoki_dt and ozoki_tc later become the OZ_DT and OZ_TC parameters.

OZ_SG is the troublesome one. As mentioned by hacker1024, the client logs a bunch of internal event details into a structure, and when initiating the login, serializes it with JSON. It then encrypts it, and the resulting string is sent as OZ_SG.

The encryption is an RC4 variant bastardized for ASCII.

OZ_TC is used as a key. The algorithm starts with h=341005, and each time a message is generated, h is translated to a textual representation and prepended to both the key and encrypted output. h is then bumped to a new value for the message.

The encryption map is then constructed, and characters in the input are then encrypted one at a time. There's some funky stuff to deal with Unicode characters, reminiscent of UTF-8 encoding, that encodes them as multiple bytes. For reference, I've included a C++ implementation of this encryption below.

class Encryptor {
    using Map = int8_t [95];
    using NumericSalt = uint32_t;
    using MessageType = std::string;  // Should be std::wstring to handle Unicode characters

    NumericSalt salt;
    Map starter_map;

    std::string makeAddedSalt() {
        NumericSalt n = salt;
        salt = salt + 2654435769 & 1048575;
        std::string text_salt;
        for (int e = 4; e > 0; e--) {
            text_salt.push_back (48 + (31 & n));
            n >>= 5;
        }
        return text_salt;
    }

public:

    Encryptor (NumericSalt start_salt = 27182818) : salt (start_salt) {
        int m = 51;
        int v = 44;
        do {
            starter_map [m] = v;
            v = (v + 88) % 95;
            m = (m + 62) % 95;
        } while (m != 51);
    }

    std::string operator() (const std::string &input_oz_salt, const MessageType &message) {
        const std::string extra_salt = makeAddedSalt();
        const std::string salt = extra_salt + input_oz_salt;

        // Initialize, then shuffle shit in u
        Map u;
        memcpy (u, starter_map, sizeof (starter_map));
        for (int v = 95, m = 94; m >= 0; m--) {
            v = (v + u [m] + salt [m % salt.length()]) % 95;
            std::swap (u [m], u [v]);
        }

        std::string encrypted;
        auto append_encrypted_char ([&encrypted, &u, m = 0, v = 0] (std::char_traits <MessageType>::int_type ch) mutable -> void {
                m = (m + 1) % 95;
                v = (v + u [m]) % 95;
                std::swap (u [m], u [v]);
                ch += u [(u [m] + u [v]) % 95];
                if (ch > 126) {
                    ch -= 95;
                }
                encrypted.push_back (ch);

        });
        for (auto ch : message) {
            if ((ch < 32 || ch > 125)) {
                // non-ascii; expand to multi-bytes as needed
                append_encrypted_char (126);
                if (ch > 2047) {
                    append_encrypted_char (80 + (ch >> 11));
                }
                append_encrypted_char (48 + (ch >> 6 & 31));
                append_encrypted_char (48 + (63 & ch));
            } else {
                append_encrypted_char (ch);
            }
        }
        return extra_salt + encrypted;
    }
};

...

Encryptor encryptor (341005); // That's the initial "h"
std::string oz_dt = encryptor (oz_tc, message);
hacker1024 commented 3 years ago

@GodStar88 This whole system is designed to block bots. I think you're out of luck.

H0r53 commented 3 years ago

Any update on this? I've been doing research as well and here are a few additional notes. As noted, OZ_SG is an encrypted JSON encoded array of messages, including various data from browser information to events. OZ_TC is used for the encryption. OZ_TC and OZ_DT come from a call to /config.json.

I've captured the unencrypted values of OZ_SG for several logins, some manual and some automated via selenium/chromium. Of course, the manual logins are successful while the automated ones are not. After looking through OZ_SG, the only thing that strikes me as a potential bot indicator are document event logs that are stored, which hold things like keystrokes and other page actions. That being said, if that was all there was to it, then you should be able to intercept the function to generate OZ_SG before encryption takes place and replace the messages with those captured from a valid / human session.

However, that doesn't seem to work. I'm thinking one of two things are happening (possibly more).

1) OZ_DT stores contextually relevant data that somehow affects the login process more than simply providing the value given by /config.json. This thread doesn't include much information on OZ_DT and what it's purpose is, so additional research may be needed there

2) Data that is sent during the /postback requests to s.hoplon.pandora.com changes what the remote end expects. In other words, by altering OZ_SG the way I mentioned earlier, it may cause a mismatch on information that was already sent

In summary, I'm trying to identify the true purpose of OZ_DT and how it is used. My original inclination was that it is a base64 encoded cryptographic hash of the config.json page (since config.json is dynamically loaded by pagespeed.js). As this data and OZ_TC change on each call to config.json, it could be that OZ_TC is used as a key for this hash. I think OZ_DT may hold some secrets in understanding what is happening.

I'm also interested in knowing why replacing (the unencrypted) OZ_SG with values from a valid previous session isn't working. Timing could be a factor. Also, it's very clear that the combination of username, password, OZ_TC, OZ_DT, and OZ_SG is a one-time-use credential. Resubmitting previously accepted values doesn't work.

hacker1024 commented 3 years ago

Also, it's very clear that the combination of username, password, OZ_TC, OZ_DT, and OZ_SG is a one-time-use credential. Resubmitting previously accepted values doesn't work.

I copied the OZ_TC, OZ_DT, and OZ_SG values from an intercepted request into my code and managed to log in with the same credentials, but not with another account.

hacker1024 commented 3 years ago

I think I missed an email where someone asked for an example of using JSON authentication to obtain a token to be used with the REST API, to be used with CURL. I can't find that message again, but I'll respond here nonetheless:

Logging in with the JSON API is a multi-step process, requiring partner authentication and sync-time handling. I can use my WIP Dart package to build a CLI tool that takes credentials and spits out an authentication token, if that's helpful to anyone? Dart can compile directly to machine code, as well as JavaScript.

H0r53 commented 3 years ago

@hacker1024 I was the one asking about that. A tool as you describe would certainly be useful.

FireController1847 commented 3 years ago

I can use my WIP Dart package to build a CLI tool that takes credentials and spits out an authentication token, if that's helpful to anyone? Dart can compile directly to machine code, as well as JavaScript.

This would be a godsend if you're able :)

hacker1024 commented 3 years ago

@H0r53 @FireController1847 Here you go: https://github.com/EpimetheusMusicPlayer/pandora_authenticator Binaries are in the releases.

FireController1847 commented 3 years ago

Sweet, thanks a ton. I managed to get the JSON endpoint up and running after quite a lot of trail and error. It appears as though the REST API does not accept the userAuthToken achieved via the JSON API. I'm not sure why that is the case so this will be super helpful

hacker1024 commented 3 years ago

I used the JSON API for the tool, but it worked for the REST API for me...

FireController1847 commented 3 years ago

I used the JSON API for the tool, but it worked for the REST API for me...

Really? Maybe I was doing something wrong, I'll take another shot at it tonight.

FireController1847 commented 3 years ago

@hacker1024 So I just took another shot at it, and it appears that the access token that I am getting through the JSON api is invalid for use with the REST api, but yours is not. What partner are you using for authentication? At the moment I am using IOS because that is the only one I could get the station.getPlaylist method to work on JSON, all the other ones failed.. The authtokens I receive are a lot shorter than yours for some reason as well. The way it is not working is it responds with "Internal server error" and a code of zero when attempting /v1/station/getStations. I haven't tried any other endpoints, but like I said it works with yours so I am not sure what could be going on.

hacker1024 commented 3 years ago

@FireController1847 I'm using Android, and I've also been able to use the old credentials I mentioned in #48. I'm not sure why your tokens are shorter, That's interesting. Could you perhaps make a new account, and provide the username, password, and an example authtoken?

FireController1847 commented 3 years ago

Sure thing. I've been using a testing account for all of this so I have no problem sharing these, I'll delete them once you're done working with them. I switched to using the same android one as you and still cannot use station.getPlaylist and it is still considered invalid for REST.

Username: [REDACTED] Password: [REDACTED] AuthToken: [REDACTED]

I will note, now that I've switched to Android the auth tokens look a lot more similar now.

hacker1024 commented 3 years ago

Interesting - that token does in fact give me the error you described. It's missing a = at the end, but that actually doesn't matter anyway as the backend can decode it just fine. I'm assuming that was just a typo.

Can you try this one: VIH3TsTsbIl6tFZQEz8DjhbTQ9rp650028pQ7q/ttfsbQI7v7N5rgEYw==

FireController1847 commented 3 years ago

Strange, the one you provided me has worked. Also, the second equal at the end was not a typo that is how I have been receiving them, I find it strange that you are receiving them differently than me haha. Maybe we should continue this discussion on a platform like Discord? If you want, of course. My username is FireController1847#3577

hacker1024 commented 3 years ago

Sure thing. @PromyLOPh, could you also maybe enable GitHub discussions?

PromyLOPh commented 3 years ago

@hacker1024 Yeah, done. Please move discussions there.

H0r53 commented 3 years ago

@PromyLOPh you provided a C++ implementation of the encryption - would it be possible to provide the corresponding decryption routine?

PromyLOPh commented 3 years ago

@H0r53 Nope, not my code, credit goes to @perette

H0r53 commented 3 years ago

Apologies, the question was intended for @perette

perette commented 3 years ago

@H0r53 Sorry, I have no plans to implement the decryption. I am curious what you need it for? With a breakpoint set in the JavaScript in the right place, you can retrieve the unencrypted messages using a browser's inspection window.

H0r53 commented 3 years ago

@perette let's say I was limited to intercepting / forwarding requests without the luxury of a browser's devtools. With decryption it would become easier to recover messages, adjust, and re-encrypt.

r0wanda commented 5 months ago

I'm a little bit late to the party, but here are some things I've noticed.