Pulkit2002 commented 3 years ago

So I accidently enrolled in Crunchyroll Beta and extension doesn't seems to be working on crunchyroll beta.

fhuhne commented 3 years ago

Thanks for opening this issue, this definitely requires some changes for the extension.

Looks like the current region is not displayed directly on the page anymore which could be a problem when I can't detect if the user actually got a valid session id.

andrei399 commented 3 years ago

I suggest looking if an anime is available to watch right now that would otherwise be unavailable (for example Attack on Titan is not available in Romania) I'm not sure if a show only available in the US exists, but my guess is there is at least one.

sabarnac commented 3 years ago

@fhuhne It appears that the website stores the country code in session storage, with the key being _ucWMConf and the value being a JSON object with a countryCode property.

Not sure if the extension would be able to access this value, but if it can, then I assume that should take care of figuring out the current region, right?

sabarnac commented 3 years ago

I tested and it appears that access to sessionStorage is indeed possible, so I guess the isUs function could be changed to something like this

function isUs() {
  // If we assume that the key which contains the JSON object with country code isn't fixed,
  // then we can just iterate through all session storage data to find it.
  // return (
  //   Object.keys(sessionStorage)
  //     .map((key) => sessionStorage[key])
  //     .map((data) => {
  //       try {
  //         return JSON.parse(data);
  //       } catch (e) {
  //         return null;
  //       }
  //     })
  //     .filter((data) => data !== null)
  //     .find((data) => "countryCode" in data)?.countryCode === "US"
  // );

  // If we assume that the key with the JSON object containing the country code will always be `._ucWMConf`,
  // then we can just do this
  return JSON.parse(sessionStorage._ucWMConf).countryCode === "US";
}

Now as for actually refreshing the token, I assume that would also have to change, and not sure what would need to be done there

Also note that this data is present in both the current and beta sites.

fhuhne commented 3 years ago

Ah, thanks for the tip. I'll test if I can make it work somehow. I got the extension so far that it at least tries to set the session id. It does not get overwritten but it looks like CR still detects the users location. I hope they are not back to checking the region on every page load ...

sabarnac commented 3 years ago

I've only managed to check till this so far. Will explore around and see if I find any requests where it gets the right region or a corrected access token at any point during navigation or playing videos

sabarnac commented 3 years ago

It appears that the beta site doesn't reload the token again after initial load. It's a React application, so it fetches the token on first load. I think for future uses of the token it might just be reading it from session storage. If the page is reloaded, then it appears to fetch a fresh token instead of re-using the one in session storage already (since now the React application is loaded from the start again). Don't know if this may cause problems. I guess the best way to figure it out is to actually get the US token and update the session storage (and maybe in other places), and see whether the updated token is used on the same React application loaded instance, as well as whether the US session persists on reloads.

Reinachan commented 3 years ago

It's been a while since there's been an update on this. Crunchyroll aim to update the web client for everyone on June 15th so it might be smart to investigate more before you can't go back to the old client anymore.

fhuhne commented 3 years ago

Sorry for the late reply. Problem is, last time I played around with it I could not find anything getting this to work again.

I'm not that much into frontend development, so I don't know if there is anything that can be done on the new app. As of now there are 2 options:

1) There is still a way and I just lack knowledge on web apps. 2) CR checks the country on each page load. They have done it before and when they are back to this mechanism there is nothing I can do.

Unfortunately I don't have enough time to get much further into 1 than I already did :(

andrei399 commented 3 years ago

So is the extension dead now that cr moved all us citizens to the beta?

Andresdiaz16 commented 3 years ago

So is the extension dead now that cr moved all us citizens to the beta

there's so much one person can do, unless we help or wait for @fhuhne to be able to fix it, if he even has the time or motivation to do so, we could consider the extension dead.

Cheos137 commented 3 years ago

Unfortunately I don't have enough time to get much further into 1 than I already did :(

I tried for some time now to get some results by fiddeling around with the crunchyroll cookies - nothing. This doesn't really mean anything tho, as I have little to no experience with web apps. It would be really nice, if you, @fhuhne could push your little progress you made ("I got the extension so far that it at least tries to set the session id"). Then we could try from that point on :)

But I have something that feels like bad news. When turning VPN off/on, only a simple (F5) page reload is required to be able to see US-only shows - not even CTRL+F5 is needed, whereas with the old page I had to close my browser, connect to VPN and then open Crunchyroll to get this to work. So it seems like Crunchyroll is checking the location each refresh and maybe (only a theory) SessionIDs are region-independant now?

andrei399 commented 3 years ago

Refreshing the page does check for session id again but ive found that when closing the connection to the vpn the us session id remains until you refresh the page so even though you go on other pages you still have the us session id

Andresdiaz16 commented 3 years ago

when closing the connection to the vpn the us session id remains until you refresh the page so even though you go on other pages you still have the us session id

Maybe they save the ID on localstorage? It would have a big impact on the page if they made a geolocalitation petition on each refresh, by printing out the contents of the localstorage I can see that they save my geolocation there as well as a UID and an Anonymous ID I'm not sure if any of this could be the session ID since I've never seen how it looks, but if the session ID is persistent it's being saved somewhere locally.

Marocco2 commented 3 years ago

when closing the connection to the vpn the us session id remains until you refresh the page so even though you go on other pages you still have the us session id

Maybe they save the ID on localstorage? It would have a big impact on the page if they made a geolocalitation petition on each refresh, by printing out the contents of the localstorage I can see that they save my geolocation there as well as a UID and an Anonymous ID I'm not sure if any of this could be the session ID since I've never seen how it looks, but if the session ID is persistent it's being saved somewhere locally.

They may have some token with TTL attached to it. Probably it's good to find a way to keep session_id value between refreshes

Cheos137 commented 3 years ago

Probably it's good to find a way to keep session_id value between refreshes

The session id is retained between all refreshes (and injecting a session id does not appear to change the region, anymore (sadly))

Cheos137 commented 3 years ago

but if the session ID is persistent it's being saved somewhere locally.

Yeah, the session id is stored inside the cookies and thus persists - it's expiry is set to 'session' tho - so browser restarts to change it, refreshes not

Andresdiaz16 commented 3 years ago

but if the session ID is persistent it's being saved somewhere locally.

Yeah, the session id is stored inside the cookies and thus persists - it's expiry is set to 'session' tho - so browser restarts to change it, refreshes not

so if injecting a new session ID doesnt change the region anymore, the only option for now would be a VPN to change the location?

Marocco2 commented 3 years ago

but if the session ID is persistent it's being saved somewhere locally.

Yeah, the session id is stored inside the cookies and thus persists - it's expiry is set to 'session' tho - so browser restarts to change it, refreshes not

then, we should look into token generation

Andresdiaz16 commented 3 years ago

but if the session ID is persistent it's being saved somewhere locally.

Yeah, the session id is stored inside the cookies and thus persists - it's expiry is set to 'session' tho - so browser restarts to change it, refreshes not

then, we should look into token generation

I'll try setting the app locally and play around with what @sabarnac suggested since from what we have seen right now the best thing would be to make it appear as we are on the US so sending the code like he suggested seems like a look onto the right path. If nothing comes up I guess we could look into the token generation.

Andresdiaz16 commented 3 years ago

So I've run some tests using a aws virtual machine, I made the change on the code as @sabarnac suggested but for some reason the extension would get completly ignored and the validation isUS wouldn't be called at all so I decided to mess around with the cookies, local storage and session storage, in localstorage I found an item that called my atention, kxgeo, this item saves the latitude and longitude where you are located as well as the provider who's making the petiton, I tried changing it with the one I got from my aws machine, but after each reload it would reset back to my current location, since this is a React app it's almost all a single page application, except for some links like the manga section or if you select original in the anime dropdown, the page won't trigger a reload, I tried changing the session_id with the one that I got from the aws machine, I tried changing the _ucWMConf too and I noticed that when I managed to get a reload on the page neither the session_id nor the _ucWMConf would change only the kxgeo would change it's value again to my current location so I think the changing the session for a US session id won't work anymore.

Cheos137 commented 3 years ago

I already thought something like that would be the case - but I wasn't able to find it. Good job doing so

Idea (idk if this could work - it may be really dumb): How bad would it be if you tunnel the initial http request to the crunchyroll servers over a cr unblocker server (much like a vpn)? I think this could work, but I'm a little concerned about the server-side as it'll maybe will get blocked by cloudflare? Anyways - just wanted to bring it out there. If it won't work for any reason - doesn't matter :)

sabarnac commented 3 years ago

There are some additional details I need to bring up.

If you look at the source of the HTML when the page loads for the first time, you'll find a script tag that sets 2 global variables __INITIAL_STATE__ and __APP_CONFIG__. The new website is a React application, which reads these global variables when the React app first runs and stores it inside a React state, then destroys the global variable.

Under __INITIAL_STATE__.global.fetchEnvironment.clientCountry you will find the country where the user is from. I've managed to get this country flag to change before the React app manages to read, but at least for me this didn't affect what results I was shown when searching for anime (shows that aren't accessible to me were still not showing up).

So I assume the session IDs/cookies are all that really matter in the end for determining which content is accessible? I guess the goal should be is to correctly set the session ID such that Crunchyroll then thinks we're in the US and changes the initial state of the application to the US region on its own. And the current way we generate session IDs doesn't appear to work I think.

The session storage variable I found can still be used to check if we're in the US region or not, but changing that and the region property under __INITIAL_STATE__ won't do anything.

sabarnac commented 3 years ago

To be more specific, the Crunchyroll website makes an API call to https://beta-api.crunchyroll.com/auth/v1/token (POST) to get the access token for all other API requests. As long as we can get this token to return a US access token, then the goal is achieved. And as far as I can see, the two headers to be concerned about that are sent to this request are the cookie header and the authorization header, which is set to Basic with an auth token whose value I cannot find where it's gotten from (it's not anywhere in session storage or cookies, nor is it in the __INITIAL_STATE__ or __APP_CONFIG__ variables that are set in the source page).

sabarnac commented 3 years ago

Andresdiaz16 commented 3 years ago

There are some additional details I need to bring up.

If you look at the source of the HTML when the page loads for the first time, you'll find a script tag that sets 2 global variables __INITIAL_STATE__ and __APP_CONFIG__. The new website is a React application, which reads these global variables when the React app first runs and stores it inside a React state, then destroys the global variable.

Under __INITIAL_STATE__.global.fetchEnvironment.clientCountry you will find the country where the user is from. I've managed to get this country flag to change before the React app manages to read, but at least for me this didn't affect what results I was shown when searching for anime (shows that aren't accessible to me were still not showing up).

So I assume the session IDs/cookies are all that really matter in the end for determining which content is accessible? I guess the goal should be is to correctly set the session ID such that Crunchyroll then thinks we're in the US and changes the initial state of the application to the US region on its own. And the current way we generate session IDs doesn't appear to work I think.

The session storage variable I found can still be used to check if we're in the US region or not, but changing that and the region property under __INITIAL_STATE__ won't do anything.

This is actually really good, maybe it could be a validation of 2 variables where it checks on the frontend for the country but it also uses the geolocation to check for the region, or something or it could be just one variable and everything changes based on that, from what I've read the Basic schema should be an arbitrary word and a column ( : ) wich it is I made the test and it always uses the same value, I dont know if this value is tied up to a location I would have to try it out on my aws machine to se if the valu changes but for now it seems its the same always, also worth mentioning this is just 1 out of 2 tokens, the second one is a JWT with the Bearer tag and this is the token contains config information about the user accout like for example if the user can have multiple streams, if it's a premium user, if it should see ads or not AND it also contains your country, also from what I've seen on the HTTP requests this is the token that they send with it so I think we are on the correct path with this we just need to figure where this json config is being set, and send to generate this token.

This is the token: imagen

And the country with the token decoded: imagen

sabarnac commented 3 years ago

To be more specific, the Crunchyroll website makes an API call to https://beta-api.crunchyroll.com/auth/v1/token (POST) to get the access token for all other API requests. As long as we can get this token to return a US access token, then the goal is achieved. And as far as I can see, the two headers to be concerned about that are sent to this request are the cookie header and the authorization header, which is set to Basic with an auth token whose value I cannot find where it's gotten from (it's not anywhere in session storage or cookies, nor is it in the __INITIAL_STATE__ or __APP_CONFIG__ variables that are set in the source page).

Oh wait, I'm an idiot. I forgot to base64 decode the Bearer token in the https://beta-api.crunchyroll.com/auth/v1/token request (thanks @Andresdiaz16 for doing what I forgot to do). I found the base64 decoded value in the source page under __APP_CONFIG__.cxApiParams.accountAuthClientId

So I guess that means we need to do 3 things on page load before the React application gets a chance to initialize:

Set the _ucWMConf.countryCode code in session storage to US
Set the __INITIAL_STATE__.global.fetchEnvironment.clientCountry in the script tag in the page source to US
Set the __APP_CONFIG__.cxApiParams.accountAuthClientId to a client ID value that would then be used by the React application to call https://beta-api.crunchyroll.com/auth/v1/token (POST) and get a access token that is valid for the US.

Alternatively, since the __APP_CONFIG__.cxApiParams.accountAuthClientId is set by Crunchyroll in the source page itself, I assume it sets this and the __INITIAL_STATE__.global.fetchEnvironment.clientCountry value based on what it receives from the cookies. So as @Andresdiaz16 points out if this is set based on location then we'd have to go with the changes suggested above. If however, these values are set purely based on what's in the cookies, then all that needs to be done is set the cookies appropriately (think this would be unlikely).

vkay94 commented 3 years ago

Set the __APP_CONFIG__.cxApiParams.accountAuthClientId to a client ID value that would then be used by the React application to call https://beta-api.crunchyroll.com/auth/v1/token (POST) and get a access token that is valid for the US.

I'm not sure whether it helps or not but I've looked into the token creation by reverse engineering the code (well, kinda - it's pretty cryptic as you probably know) and I found the following snippet:

function D() {
    const {
        anonClientId: e,
        accountAuthClientId: t
    } = s.cxApiParams, r = {
        headers: {
            Authorization: `Basic ${window.btoa(`${e}:`)}`
        },
        data: "grant_type=client_id"
    }, n = {
        headers: {
            Authorization: `Basic ${window.btoa(`${t}:`)}`
        },
        data: "grant_type=etp_rt_cookie"
    };
    return (e, t, {
        API: o
    }) => {
        const i = () => o.Auth.createToken({}, r).then((t => (e(j(u.sk)), t))),
            c = () => o.Auth.createToken({}, n).then((t => (e(j(u.NG)), t)));
        return (0, a.TF)(t()) ? i() : (0, a.Y)(t()) ? c() : c().catch((() => i()))
    }
}

While looking into the call info via Firefox Dev Console the token call passes grant_type=etp_rt_cookie in its request header. According to the above snippet the required data for this type is t which is accountAuthClientId. So it seems like you guessed.

Marocco2 commented 3 years ago

I've done some reverse engineering to the send token request. I'm not sure if it's useful or not, neither if it's accurate. I'll post it below

30803: (e, t, r) =>{
    'use strict';
    var checker_class = r(69814),
    o = r(96145),
    s = r(47970),
    i = r(79537),
    a = r(96415),
    u = r(59210),
    c = r(48870),
    maybe_console_logger = r(1511);
    e.exports = function (maybe_main_class) {
      return new Promise((function (t, jump_to_function) {
        var request_token = maybe_main_class.data,
        request_token_header = maybe_main_class.headers;
        checker_class.isFormData(request_token) && delete request_token_header['Content-Type'];
        var http_request_token_class = new XMLHttpRequest;
        if (maybe_main_class.auth) {
          var account_username_string = maybe_main_class.auth.username || '',
          account_password_string = maybe_main_class.auth.password ? unescape(encodeURIComponent(maybe_main_class.auth.password)) : '';
          request_token_header.Authorization = 'Basic ' + btoa(account_username_string + ':' + account_password_string)
        }
        var y = a(maybe_main_class.baseURL, maybe_main_class.url);
        if (http_request_token_class.open(maybe_main_class.method.toUpperCase(), i(y, maybe_main_class.params, maybe_main_class.paramsSerializer), !0), http_request_token_class.timeout = maybe_main_class.timeout, http_request_token_class.onreadystatechange = function () {
          if (http_request_token_class && 4 === http_request_token_class.readyState && (0 !== http_request_token_class.status || http_request_token_class.responseURL && 0 === http_request_token_class.responseURL.indexOf('file:'))) {
            var getAllResponseHeaders = 'getAllResponseHeaders' in http_request_token_class ? u(http_request_token_class.getAllResponseHeaders()) : null,
            request_template = {
              data: maybe_main_class.responseType && 'text' !== maybe_main_class.responseType ? http_request_token_class.response : http_request_token_class.responseText,
              status: http_request_token_class.status,
              statusText: http_request_token_class.statusText,
              headers: getAllResponseHeaders,
              config: maybe_main_class,
              request: http_request_token_class
            };
            o(t, jump_to_function, request_template),
            http_request_token_class = null
          }
        }, http_request_token_class.onabort = function () {
          http_request_token_class && (jump_to_function(maybe_console_logger('Request aborted', maybe_main_class, 'ECONNABORTED', http_request_token_class)), http_request_token_class = null)
        }, http_request_token_class.onerror = function () {
          jump_to_function(maybe_console_logger('Network Error', maybe_main_class, null, http_request_token_class)),
          http_request_token_class = null
        }, http_request_token_class.ontimeout = function () {
          var log = 'timeout of ' + maybe_main_class.timeout + 'ms exceeded';
          maybe_main_class.timeoutErrorMessage && (log = maybe_main_class.timeoutErrorMessage),
          jump_to_function(maybe_console_logger(log, maybe_main_class, 'ECONNABORTED', http_request_token_class)),
          http_request_token_class = null
        }, checker_class.isStandardBrowserEnv()) {
          var header_name = (maybe_main_class.withCredentials || c(y)) && maybe_main_class.xsrfCookieName ? s.read(maybe_main_class.xsrfCookieName) : void 0;
          header_name && (request_token_header[maybe_main_class.xsrfHeaderName] = header_name)
        }
        if ('setRequestHeader' in http_request_token_class && checker_class.forEach(request_token_header, (function (e, content_type) {
          void 0 === request_token && 'content-type' === content_type.toLowerCase() ? delete request_token_header[content_type] : http_request_token_class.setRequestHeader(content_type, e)
        })), checker_class.isUndefined(maybe_main_class.withCredentials) || (http_request_token_class.withCredentials = !!maybe_main_class.withCredentials), maybe_main_class.responseType) try {
          http_request_token_class.responseType = maybe_main_class.responseType
        } catch (maybe_some_error) {
          if ('json' !== maybe_main_class.responseType) throw maybe_some_error
        }
        'function' == typeof maybe_main_class.onDownloadProgress && http_request_token_class.addEventListener('progress', maybe_main_class.onDownloadProgress),
        'function' == typeof maybe_main_class.onUploadProgress && http_request_token_class.upload && http_request_token_class.upload.addEventListener('progress', maybe_main_class.onUploadProgress),
        maybe_main_class.cancelToken && maybe_main_class.cancelToken.promise.then((function (e) {
          http_request_token_class && (http_request_token_class.abort(), jump_to_function(e), http_request_token_class = null)
        })),
        request_token || (request_token = null),
        http_request_token_class.send(request_token)
      }))
    }
  },

Andresdiaz16 commented 3 years ago

I haven't been able to look further, between college and work, @sabarnac could you walk us thru on how you managed to change the country before the react apps fully loads up? that's gonna be really helpful on my tests and would save me a lot of time, @Marocco2 and @vkay94 I think both codes you managed to pull could be really helpful further if getting a US country becomes more than just changing some varibles like we are expecting.

vkay94 commented 3 years ago

I looked deeper into the token request the last day and unfortunately I assume the country info which is used for this request is the access_token's payload - it also contains a country key (it can be tested with a JWT decoder).

I've tried to intercept the request by using the webRequest API and StreamFilter: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/StreamFilter/ondata

The interception did work because if the access_token key was changed or deleted I was logged out. If the `country key was deleted nothing changed. I can't tell whether it applies to not logged in users because I'm not in the US.

So changing the country key doesn't work (at least for the way I tried).

I've used this snippet in background_script.js (+ deleting the other content):

var target = "https://beta-api.crunchyroll.com/auth/v1/token";

function listener(details) {

  let filter = browser.webRequest.filterResponseData(details.requestId);
  let decoder = new TextDecoder("utf-8");
  let encoder = new TextEncoder();

  let data = [];
  filter.ondata = event => {
    data.push(event.data);
  }

  filter.onstop = event => {
    let str = "";
    for (let buffer of data) {
        str += decoder.decode(buffer, {
            stream: true
        });
    }
    str += decoder.decode(); // end-of-stream

    var jsonData = JSON.parse(str)

    delete jsonData.country

    filter.write(encoder.encode(JSON.stringify(jsonData)));
    filter.close();
  };
}

browser.webRequest.onBeforeRequest.addListener(
  listener, {
  urls: [target]
  }, ['blocking']
);

Good luck to the others, probably you'll find something to work on ;)

fhuhne commented 3 years ago

Hey everyone,

just wanted to hop in for a sec and say that I'm very thankful for all your effort so far! Unfortunately I'm still too busy to invest any meaningful time here so right now this project depends on you guys. But I'm still here and watching the progress, so when a way is found I'll update the extension.

Again, thank you all and good luck finding a way 😀

tomo0611 commented 3 years ago

I've done watching videos without reverse engineering or VPNs. However the script is not good. I'll post it below

The processes are consists of 4 steps.

Detect https://beta-api.crunchyroll.com/auth/v1/token request.
Open google script page to exec (due to CROS), read it and save it on memory.
Second time and after requests, it injects US token json to token request.
You can watch videos without VPNs!!

To be sloved

Currently the auth header and token are fixed and hardcoded in the script, That should be sholved. (Send cookie and auth header as a param will be good.
Opening a new tab is annoying. (Solve CROS problem.)
Non-Logined and Logined request are different. It should support both of them.

Screenshots

Screenshot 2021-08-18 111919

Screenshot 2021-08-18 112044

Screenshot 2021-08-18 112243

Code

background.js

Of course, add some domains related to GAS to manifest.json is required.

var token_data = "{}";

function onCreated(tab) {
    console.log(`Created new tab: ${tab.id}`)
}

function onError(error) {
    console.log(`Error: ${error}`);
}

function getUSToken() {
    var creating = browser.tabs.create({
        url:"https://script.google.com/macros/s/XXXXXXX/exec"
    });
    creating.then(onCreated, onError);
}

function listener(details) {    
  let filter = browser.webRequest.filterResponseData(details.requestId);
  let decoder = new TextDecoder("utf-8");
  let encoder = new TextEncoder();
  let data = [];

  filter.ondata = event => {
    data.push(event.data);
  }

  filter.onstop = event => {
    let str = "";
    for (let buffer of data) {
        str += decoder.decode(buffer, {
            stream: true
        });
    }
    str += decoder.decode(); // end-of-stream
    console.log(JSON.stringify(JSON.parse(token_data), null, 4));
    if(token_data=="{}"&&details.url=="https://beta-api.crunchyroll.com/auth/v1/token"){
        getUSToken();
    }
    if(details.url.startsWith("https://script.googleusercontent.com/macros/echo?user_content_key=")){
        token_data = str;
        filter.write(encoder.encode(str));
    } else if(token_data==="{}"){
        console.log("token data not found! Normal Response");
        filter.write(encoder.encode(str));
    } else{
        console.log("token data found! Give token data");
        filter.write(encoder.encode(token_data));
    }
    filter.close();
  };

  return {};
}

browser.webRequest.onHeadersReceived.addListener(
  listener, {
  urls: ["https://beta-api.crunchyroll.com/auth/v1/token","https://script.googleusercontent.com/macros/echo?user_content_key=*"]
  }, ['blocking','responseHeaders']
);

GAS (Google Apps Script)

function doGet(e) {

  var options = {
    method: "post",
    headers:{
      "Authorization": "Basic XXXXXXX",
      "Accept-Language": "en-US",
      "Content-Type": "application/x-www-form-urlencoded",
      "Cookie": "etp_rt=XXX-XXXX-XXX-XXX-XXXX;",
      "Origin": "https://beta.crunchyroll.com",
…  console.log("Response code is "+code);
  var responseBody = response.getContentText();
  Logger.log(responseBody);

  return ContentService
  .createTextOutput(responseBody)
  .setMimeType(ContentService.MimeType.Html);
}

Andresdiaz16 commented 3 years ago

@tomo0611 This is actually huge, we could actually take your approach and just tweak it so,:

First, it doesn't utilize the google apps script or a way so that opening a new tab is not needed.
Two, maybe we could use the CR-Unblocker to get a US token, and we should need to have it hard codes just replace the new auth an cookie values.
The differnt implementations between currently logged in and not logged it accouts could be looked on the future or as we go looking for a new solution.

Thanks again everyone for taking your time in this, I haven't been able to look into this project that much, beteen uni, work and limited knowledge in web development there isn't much time left for me to look into personal projects. I'll try to get this going again,

EsmailELBoBDev2 commented 3 years ago

Any updates? i absolutely suck when it comes to NPM so any ideas or steps how to impelement these steps?

Andresdiaz16 commented 3 years ago

Any updates? i absolutely suck when it comes to NPM so any ideas or steps how to impelement these steps?

Yes, just not a good one, so as far as I've seen the main thing we need to change is the token petition, in this somewhere in a Json your Countries ID is sent, and this generates a JWT, this token is later used in every single petiton made to the beta api, so for starters we have two options here:

First, we could try to intercept the petition before its made, change the Country to US and send it, that way in theory we would receive a JWT with a US tag that would allow us to watch anime like we where on the US.
Or the second option and the least efficient is to get a US token and change the Authorization header each time a petition is made, this is not the best solution, it would add some delay time to the petition depending on your connection.

All of this is made on the assumption that we only need to change the Json object "Country", im really inexperienced when it comes to extensions development and web development, further more I study and work so that leaves me with around 2 or 3 hours a day to work on this, sometimes not even that so progress is slow, but I have hope we can pull this off, many here have made great discoveries towards that goal, you could start by looking at how to develop extensions if you want to implement what @tomo0611 did, or look at the page petition and code to help with the investigation, it doesn't matter if it's something big or just a variables name you found any help is welcome and appreciated.

I'll try to keep this thread updated if I manage to find something else or the progress I make.

insidewhy commented 3 years ago

I've been building on the work of @tomo0611 , I'm an experienced full-stack developer so this is something I should be able to easily handle. Will still need a US server that isn't blocked by crunchyroll though.

~Unfortunately it seems that web extensions are not allowed to access the Authorization header so I don't think there's any way to get this to work automatically (not unless I can alter the code/html of the website itself via the extension to grab the authorisation header from outside of the web request listener). The best I can do so far is issue a message to prompt the user to use the network console to grab the authorisation header themselves and instruct them to paste it to a GAS that will then request the token on their behalf. Of course this gets pretty annoying, but at least it works. The authorisation header is fixed so it's possible to make this a configuration parameter of the extension.~

insidewhy commented 3 years ago

All of this is made on the assumption that we only need to change the Json object "Country"

@Andresdiaz16 Sorry this can never work. Sure I can pull apart the JWT easily and hack the "Country" but then when I reconstruct the JWT the signature won't match. The only way to regenerate the signature is to use crunchyroll's private key, but of course this isn't something they share since this is essential to the security of their site. So as soon as the modified JWT is sent back to crunchyroll they will detect it has been modified and reject it.

Marocco2 commented 3 years ago

All of this is made on the assumption that we only need to change the Json object "Country"

@Andresdiaz16 Sorry this can never work. Sure I can pull apart the JWT easily and hack the "Country" but then when I reconstruct the JWT the signature won't match. The only way to regenerate the signature is to use crunchyroll's private key, but of course this isn't something they share since this is essential to the security of their site. So as soon as the modified JWT is sent back to crunchyroll they will detect it has been modified and reject it.

So the most feasible option for us is to trick CR to sign the JWT with country US

insidewhy commented 3 years ago

So the most feasible option for us is to trick CR to sign the JWT with country US

As @tomo0611 discovered, this will involve intercepting the token request and then sending it from a US based server. ~To automate this I need an alternative mechanism for finding the authorisation header since web extensions are not allowed to access it via browser.webRequest.onHeadersReceived.addListener. This could be hard, or even impossible.~ Just found out the authorisation header isn't even a secret and is the same for all users... it's bm9haWhkZXZtXzZpeWcwYThsMHE6.

insidewhy commented 3 years ago

Here's a minimal curl example to request the token:

curl -X POST -d grant_type=etp_rt_cookie \
  -H "Authorization: Basic bm9haWhkZXZtXzZpeWcwYThsMHE6" \
  -H "Cookie: etp_rt=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" \
  https://beta-api.crunchyroll.com/auth/v1/token

insidewhy commented 3 years ago

~I'm pretty sure that the authorization header is fixed to the account and doesn't change, so it's reasonable that you could enter that into the extension.~ I've just discovered that the authorisation token is fixed for all users, it is bm9haWhkZXZtXzZpeWcwYThsMHE6. The etp_rt cookie changed when I logged in with a different browser. ~Not sure if it expires, if not then that could be configured too.~ No need to configure that, think it can easily be read from the cookies by the extension.

NiceArgie commented 2 years ago

Hey guys, I wanted to know if the solution you found still works. I've been using this extension for a long time, and even though I know nothing of coding, I've been able to make the extension work up until crunchyroll started forcing the beta. I've read the whole thread on this issue but I literally have no idea how to do anything you said haha. If someone could help me understand how to set up the extension to work, I'd really appreciated, but if you can't, thanks regardless.

insidewhy commented 2 years ago

@NiceArgie You could try using https://github.com/hyugogirubato/Kamyroll-Python, I added support for socks proxies to it, then I create a socks proxy over ssh using ssh -D 1080 <host> where <host> is a VPS provider in the US that is not on crunchyroll's restricted list. Many VPS won't work (i.e. digital ocean) but some will.

Ashesh3 commented 2 years ago

@tomo0611 Thank you! Your method works flawlessly on the beta site.

manifest.json for anyone using the method:

{
  "manifest_version": 2,
  "name": "Beta CR-Unblocker",
  "version": "2.3.2",
  "description": "A tool that will set the Crunchyroll Session ID to an American session ID",
  "homepage_url": "https://github.com/fhuhne/CR-Unblocker/",
  "author": "fhuhne",
  "permissions": [
    "activeTab",
    "cookies",
    "blocking",
    "webRequestBlocking",
    "responseHeaders",
    "notifications",
    "storage",
    "webRequest",
    "*://*.crunchyroll.com/*",
    "*://*.cr-unblocker.us.to/*",
    "*://*.googleusercontent.com/*"
  ],
  "background": {
    "scripts": ["background_script.js"]
  }
}

GAS (Google Apps Script)

function doGet(e) {

  var options = {
    method: "post",
    headers:{
      "Authorization": "Basic XXXXXXX",
      "Accept-Language": "en-US",
      "Content-Type": "application/x-www-form-urlencoded",
      "Cookie": "etp_rt=XXX-XXXX-XXX-XXX-XXXX;",
      "Origin": "https://beta.crunchyroll.com",
      "Referer": "https://beta.crunchyroll.com/",
      "User-Agent": "Mozilla/5.9 (Windows NT 19.6; Win64; x64; rv:93.6) Gecko/26198161 Firefox/93.6",
    },
  payload : "grant_type=etp_rt_cookie",
  muteHttpExceptions: true,
  escaping: false
  }

  var response = UrlFetchApp.fetch("https://beta-api.crunchyroll.com/auth/v1/token", options);
  var code = response.getResponseCode();
  console.log("Response code is "+code);
  var responseBody = response.getContentText();
  Logger.log(responseBody);

  return ContentService
  .createTextOutput(responseBody)
  .setMimeType(ContentService.MimeType.Html);
}

Ashesh3 commented 2 years ago

Created an updated addon with a new working method, for Crunchyroll Beta: https://github.com/Ashesh3/CR-Unlocker-Reborn

Atemu commented 2 years ago

If you’ve followed a link from another site for an extension or theme, that item is no longer available.

Ashesh3 commented 2 years ago

If you’ve followed a link from another site for an extension or theme, that item is no longer available.

The addon is waiting a review from the Mozilla store and will be live shortly. Check back in sometime. Edit: seems like its taking longer than usual for them.. Maybe because it's a weekend.

fhuhne commented 2 years ago

Wow, it seems the complexity of your update went down to an unbelievable minimum!

Ashesh3 commented 2 years ago

@Atemu extension is approved, you can install and test it now.

fhuhne / CR-Unblocker

Not Working with CrunchyRoll Beta #5

The processes are consists of 4 steps.

To be sloved

Screenshots

Code

background.js

GAS (Google Apps Script)

GAS (Google Apps Script)