UltimaHoarder / UltimaScraper

Scrape all the media from an OnlyFans account - Updated regularly
GNU General Public License v3.0
3.88k stars 610 forks source link

Session Lock | Please refresh the page error #1078

Closed Macmasteri closed 3 years ago

Macmasteri commented 3 years ago

Running the latest version and got this error;

Auth (V1) Attempt 1/10 Please refresh the page Auth (V1) Attempt 2/10 Please refresh the page Auth (V1) Attempt 3/10 Please refresh the page Auth (V1) Attempt 4/10 Please refresh the page Auth (V1) Attempt 5/10 Please refresh the page Auth (V1) Attempt 6/10 Please refresh the page Auth (V1) Attempt 7/10 Please refresh the page Auth (V1) Attempt 8/10 Please refresh the page Auth (V1) Attempt 9/10 Please refresh the page Auth (V1) Attempt 10/10 Please refresh the page

Whatsmyname22 commented 3 years ago

I'm having the same problem

Macmasteri commented 3 years ago

I'm having the same problem

Worked great during the day but suddenly this appeard. Tried "refresh" and a another account as well with no luck.

sonicshake commented 3 years ago

Same problem here, been using the script for ~9 months and never had auth issues until now.

sonicshake commented 3 years ago

In config.json "browser": { "auth" } seems to be undocumented but I tried changing it to false anyway in case this is a problem with forcing browser authentication. Doesn't seem to have helped though.

UltimaHoarder commented 3 years ago

OF changed their authentication, I'll fix it.

UltimaHoarder commented 3 years ago

Will update the script within 2 hours.

NovaResonance commented 3 years ago

Will update the script within 2 hours.

Thanks for the heads up and the speediness of working on it

UltimaHoarder commented 3 years ago

Reverse engineering takes awhile .-.

I'm still working on it, but I'm making good progress.

This is what I have to unravel lmao

image

oftrash commented 3 years ago

Thanks for looking into this.

btkador commented 3 years ago

was just digging into it and ended up at the same code snipped in vendor.js you've posted the screenshot from.. so seems I'm on the right track.

I've worked on a similar issue with the cloudflare captchas and the best solution there was copy/extract their relevant javascript functions and execute them to get the token - instead of decoding the function. That way - if they change something - you just execute the change they made and always get the correct token. It worked like.. fetch the current js.. extract the function with regex.. execute the function and get the token.

Hope that helps :-)

zivshek commented 3 years ago

Noticed there is no auth_hash anymore in the cookie, lol.

Btw, impressive reverse engineering right there. Compared to this, Recurbate is just too easy.

Macmasteri commented 3 years ago

I gotta say very fast response on this @DIGITALCRIMINAL Much appreciated! Huge fan on onlyfans specially TS :P

Will there be a new version uploaded?

RonnieBlaze commented 3 years ago

how much time will it take? im stuck lol, nvm i'll try stuckporn

HEY PLS FIX IT!

it will be fixed when its fixed,.

nappyapp commented 3 years ago

how much time will it take? im stuck lol, nvm i'll try stuckporn

HEY PLS FIX IT!

Be the change you wish to see in the world

hackerin0 commented 3 years ago

So is this the end? Did OF successfully block people from downloading the stuff they pay for?

btkador commented 3 years ago

guys, this is not a reddit thread. let DIGITALCRIMINAL work, and stop complaining.

pay him a donation for his work instead or post constructive help instead of wining.

oftrash commented 3 years ago

been looking into this also. looks like OF added 3 new fields (x-bc / sign and time).

To help @DIGITALCRIMINAL out my sign is 3:XXXX:XXX:608c48da

I have logged into about 10 different account and it looks like the sign always starts with a 3:, 3 characters for the 3rd value and 608c48da for the last value.

Macmasteri commented 3 years ago

This scraping method is a fantastic tool and I have no idea how tough the programming skills is for this one, I only raise DIGITALCRIMINAL to the skys for sharing his wonderful stuff for us. DIGITAL CRIMINAL is probably dealing with this right now, let him do his work and stay tuned.

UltimaHoarder commented 3 years ago

sign is required or you'll get "please refresh" error. sign's sha1 hash is made up of:

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO\n1619875069\n/api2/v2/users/me\n0"

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO" never changes (static). "1619875069" is the time "/v2/users/me" is the url path. You also have to include the params if there are any. "0" I don't know what this is, but it's part of a "or" statement and it usually defaults to this.

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO\n1619875069\n/api2/v2/users/me\n0" gets hashed into sha1.

sha1 result: 5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2

It then becomes part of sign.

3:5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2:8bc:608c48da

3: is static 5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2: isn't static 8bc: isn't static 608c48da is static

8bc - this part seems to be the final problem and if you substitute anything for it, you'll get "please refresh" 8bc is calculated by the whole math function I posted above. the sha1 hashed is used in it's calculations which is why it will fail if you change it. ###################### x-bc is required or you'll get your session locked and reset. It's similar to "sess".

You need it to make any requests to their API. ###################### Technically, the sign function should be like this... but the "a55" part in "final_sign" is incorrect.

def create_sign(session, link, sess, user_agent, text="onlyfans"):
    # Users: 300000 | Creators: 301000
    # time2 = str(int(round(time.time() * 1000-301000)))
    session.headers["access-token"] = sess

    def btoa(specimen):
        encoded = base64.b64encode(str(specimen).encode('ascii'))
        encoded = encoded.decode("utf-8")
        return encoded
    # Create sign
    time2 = str(int(round(time.time())))
    path = urlparse(link).path
    query = urlparse(link).query
    path = path if not query else f"{path}?{query}"
    static_param = "BcsLYSyemJCNrob8u6QWziudT5Xx4LlO"
    a = [static_param, time2, path, "0"]
    msg = "\n".join(a)
    message = msg.encode("utf-8")
    hash_object = hashlib.sha1(message)
    sha_1_sign = hash_object.hexdigest()
    final_sign = f"3:{sha_1_sign}:a55:608c48da"
    # Create x-bc - You don't need to do this, infact I don't think you can create a customer token, you'll have to use it from the browser.
    t = 1e12
    encoded_time = btoa(time2)
    random_math0 = random()
    encoded_random_math0 = btoa(random_math0*t)
    random_math1 = random()
    encoded_random_math1 = btoa(random_math1*t)
    encoded_ua = btoa(user_agent)
    temp_string = f"{encoded_time}.{encoded_random_math0}.{encoded_random_math1}.{encoded_ua}"
    final_string = temp_string.encode("utf-8")
    hash_object = hashlib.sha1(final_string)
    sha_1_xbc = hash_object.hexdigest()
    # 
    session.headers["sign"] = final_sign
    session.headers["time"] = time2
    session.headers["x-bc"] = "YOUR X-BC TOKEN"
    return session

This is the old sign function that I made. https://github.com/DIGITALCRIMINAL/OnlyFans/blob/b0cf772a83ebe9adaa232a683036ea8b8cd288a7/apis/onlyfans/onlyfans.py#L19

Macmasteri commented 3 years ago

sign is required or you'll get "please refresh" error. sign's sha1 hash is made up of:

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO\n1619875069\n/api2/v2/users/me\n0"

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO" never changes (static). "1619875069" is the time "/v2/users/me" is the url path. You also have to include the params if there are any. "0" I don't know what this is, but it's part of a "or" statement and it usually defaults to this.

"BcsLYSyemJCNrob8u6QWziudT5Xx4LlO\n1619875069\n/api2/v2/users/me\n0" gets hashed into sha1.

sha1 result: 5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2

It then becomes part of sign.

3:5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2:8bc:608c48da

3: is static 5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2: isn't static 8bc: isn't static 608c48da is static

8bc - this part seems to be the final problem and if you substitute anything for it, you'll get "please refresh" 8bc is calculated by the whole math function I posted above. the sha1 hashed is used in it's calculations which is why it will fail if you change it. ###################### x-bc is required or you'll get your session locked and reset. It's similar to "sess".

You need it to make any requests to their API. ###################### Technically, the sign function should be like this... but the "a55" part in "final_sign" is incorrect.

def create_sign(session, link, sess, user_agent, text="onlyfans"):
    # Users: 300000 | Creators: 301000
    # time2 = str(int(round(time.time() * 1000-301000)))
    session.headers["access-token"] = sess

    def btoa(specimen):
        encoded = base64.b64encode(str(specimen).encode('ascii'))
        encoded = encoded.decode("utf-8")
        return encoded
    # Create sign
    time2 = str(int(round(time.time())))
    path = urlparse(link).path
    query = urlparse(link).query
    path = path if not query else f"{path}?{query}"
    static_param = "BcsLYSyemJCNrob8u6QWziudT5Xx4LlO"
    a = [static_param, time2, path, "0"]
    msg = "\n".join(a)
    message = msg.encode("utf-8")
    hash_object = hashlib.sha1(message)
    sha_1_sign = hash_object.hexdigest()
    final_sign = f"3:{sha_1_sign}:a55:608c48da"
    # Create x-bc - You don't need to do this, infact I don't think you can create a customer token, you'll have to use it from the browser.
    t = 1e12
    encoded_time = btoa(time2)
    random_math0 = random()
    encoded_random_math0 = btoa(random_math0*t)
    random_math1 = random()
    encoded_random_math1 = btoa(random_math1*t)
    encoded_ua = btoa(user_agent)
    temp_string = f"{encoded_time}.{encoded_random_math0}.{encoded_random_math1}.{encoded_ua}"
    final_string = temp_string.encode("utf-8")
    hash_object = hashlib.sha1(final_string)
    sha_1_xbc = hash_object.hexdigest()
    # 
    session.headers["sign"] = final_sign
    session.headers["time"] = time2
    session.headers["x-bc"] = "YOUR X-BC TOKEN"
    return session

This is the old sign function that I made. https://github.com/DIGITALCRIMINAL/OnlyFans/blob/b0cf772a83ebe9adaa232a683036ea8b8cd288a7/apis/onlyfans/onlyfans.py#L19

So for a total rookie in this. Is there a way around it to get the scraper running again? :)

LunarPenguin commented 3 years ago

for the scraper to run we need to login to OF for the scraper to login to OF we need the signing function to work, rinse and repeat

Macmasteri commented 3 years ago

for the scraper to run we need to login to OF for the scraper to login to OF we need the signing function to work, rinse and repeat

Ok thanks :)

ghost commented 3 years ago

@DIGITALCRIMINAL Sir, can you try to make it to login with cookies.txt file that we can get through one extension? i think through cookies,txt file we grab from site can be use to login and get content from the site

i am still new but please think once sir

oftrash commented 3 years ago

@DIGITALCRIMINAL Sir, can you try to make it to login with cookies.txt file that we can get through one extension? i think through cookies,txt file we grab from site can be use to login and get content from the site

i am still new but please think once sir

not possible to do any of this by cookies, We need to figure out how to 100% create the sign request param.

LunarPenguin commented 3 years ago

@thehenil what people need to remember is that the cookies for the most part are static values (that means they change very little over time, except for browser upgrades, site changes or logouts) the sign value we are talking about changes for every request we do so it is not possible to set it statically

oftrash commented 3 years ago

FYI I am willing to drop $2k to venmo,paypal,cash.app if someone can figure out the full sign

ghost commented 3 years ago

Can someone give me more background information? I want to understand the challenge behind this. My understanding is that the sign is created on client side. Doesn't that mean there has to be a javascript function that computes this sign and that can be used to understand how the sign is computed?

LunarPenguin commented 3 years ago

yes, there is a client side function, but we need to reverse engineer the function to determine how to generate our own sign value for the requests we want to do. and obfuscated javascript does not make your life any easier either

btkador commented 3 years ago

@nezrok it's hardcore obfuscated so not easy to reproduce

image

you can find it in vendor.js if you search for "869c"

oftrash commented 3 years ago

@nezrok it's hardcore obfuscated so not easy to reproduce

image

you can find it in vendor.js if you search for "869c"

Just to make it easier here is vendor.js https://static.cdn.onlyfans.com/theme/onlyfans/spa/vendor.js?rev=202104301813-271ad07a42

I am using https://codebeautify.org/jsviewer to at least not go bat shit crazy looking at this obfuscated code

der-pepe commented 3 years ago

So, I know you're working very hard and cracking the function will be plainly awesome (until they mess things up again, which they most definitelly will) but, as mentioned above , doing a js->py sintax re-format and feeding it into an exec/eval isn't feasible? just wondering...

LunarPenguin commented 3 years ago

@DIGITALCRIMINAL i would like to suggest you take a look at JSNICE it attempts to give the random vars readable names

UltimaHoarder commented 3 years ago

@DIGITALCRIMINAL i would like to suggest you take a look at JSNICE it attempts to give the random vars readable names

I did that, but it didn't help

UltimaHoarder commented 3 years ago

So, I know you're working very hard and cracking the function will be plainly awesome (until they mess things up again, which they most definitelly will) but, as mentioned above , doing a js->py sintax re-format and feeding it into an exec/eval isn't feasible? just wondering...

I mean yeah, but there's too many variables it uses. We could just use selenium, but that requires more memory, and I don't think it can easily be threaded.

btkador commented 3 years ago

I mean yeah, but there's too many variables it uses. We could just use selenium, but that requires more memory, and I don't think it can easily be threaded.

probably use https://github.com/PiotrDabkowski/Js2Py and just execute the function? selenium is way to heavy

ghost commented 3 years ago

So, I know you're working very hard and cracking the function will be plainly awesome (until they mess things up again, which they most definitelly will) but, as mentioned above , doing a js->py sintax re-format and feeding it into an exec/eval isn't feasible? just wondering...

I mean yeah, but there's too many variables it uses. We could just use selenium, but that requires more memory, and I don't think it can easily be threaded.

What about just downloading the cookies with something like this https://addons.mozilla.org/nl/firefox/addon/cookies-txt/ And then loading it into a request session like this:

def parse_cookie_file(cookie_file):
    cookies = {}
    with open(cookie_file, "r") as fp:
        for line in fp:
            if not re.match(r"^#", line):
                line_fields = line.strip().split("\t")
                cookies[line_fields[5]] = line_fields[6]
    return cookies

session = requests.session()
cookies = parse_cookie_file(cookie_file)
session.cookies.update(cookies)
der-pepe commented 3 years ago

So, I know you're working very hard and cracking the function will be plainly awesome (until they mess things up again, which they most definitelly will) but, as mentioned above , doing a js->py sintax re-format and feeding it into an exec/eval isn't feasible? just wondering...

I mean yeah, but there's too many variables it uses. We could just use selenium, but that requires more memory, and I don't think it can easily be threaded.

I didn't meant to translate the whole thing, mainly just create the native functions for the arithmetic operators of the "r" object (later "n"), and then doing a regex replace of the "return Math[r(..." contents. It probably needs some extra substitutions and tweaks, so it might end up in another deep rabbit hole, anyway... I'm not very python savvy, but I think I'm going to dust off the VSCode and take a look...

mwight4 commented 3 years ago

One quick thought that just hit me, its in the vendor file so that "normally" means its an external library is there anyway to search for some of the strings online and find the library they are using

der-pepe commented 3 years ago

So, I know you're working very hard and cracking the function will be plainly awesome (until they mess things up again, which they most definitelly will) but, as mentioned above , doing a js->py sintax re-format and feeding it into an exec/eval isn't feasible? just wondering...

I mean yeah, but there's too many variables it uses. We could just use selenium, but that requires more memory, and I don't think it can easily be threaded.

I didn't meant to translate the whole thing, mainly just create the native functions for the arithmetic operators of the "r" object (later "n"), and then doing a regex replace of the "return Math[r(..." contents. It probably needs some extra substitutions and tweaks, so it might end up in another deep rabbit hole, anyway... I'm not very python savvy, but I think I'm going to dust off the VSCode and take a look...

Nevermind, it is quite the sinkhole... lots of nested functions with morphed parameters, I can't think of any way other than code-by-code replacement, starting from that "Return" and going step by step to the outside... good luck man, and kudos for your work!

hackerin0 commented 3 years ago

What about using Python Selenium with (headless?) firefox or chrome? I made many scrapers with that, I also managed to run it with multiple threads. Performance is not the best, but if it's the only way around.... That will work for sure

jumbubly commented 3 years ago

DC is working on the code, it will take time because of reverse engineering so please hope for the best and wait a while, patience is virtue.

oftrash commented 3 years ago

Just to make this clear we just need the 3rd value of the sign. in this case 75e Right?

sign: 3:XXXXXXXX:75e:608c48da

UltimaHoarder commented 3 years ago

Just to make this clear we just need the 3rd value of the sign. in this case 75e Right?

sign: 3:XXXXXXXX:75e:608c48da

Yes, we need to be able to create that value ourselves

oftrash commented 3 years ago

Just to make this clear we just need the 3rd value of the sign. in this case 75e Right? sign: 3:XXXXXXXX:75e:608c48da

Yes, we need to be able to create that value ourselves

Yep, I am just trying to steer this topic back to what it sould be since most people are not really reading the issue!

cariah commented 3 years ago

Is there a discord or chat somewhere we can discuss our findings?

hippothon commented 3 years ago

If the dude offering money is serious my email is in my profile. This is what I'm using in my code. Javascript but you can port it to python easily enough.

I don't know if the example above (5b0e330fcdbf9507c4391e6c2a9e4a98d3c900f2 and 8bc) was intentionally incorrect or if I typoed something while cleaning up the code giving random errors since I get 8bd instead, but I've been running this for about a day with no issues.

var check = (hash) => {
  return Math.abs(
    hash.charCodeAt(15) - 128 +
    hash.charCodeAt(3) + 127 +
    hash.charCodeAt(27) + 141 +
    hash.charCodeAt(38) + 82 +
    hash.charCodeAt(31) - 126 +
    hash.charCodeAt(23) + 93 +
    hash.charCodeAt(4) + 102 +
    hash.charCodeAt(35) - 108 +
    hash.charCodeAt(9) + 71 +
    hash.charCodeAt(25) - 130 +
    hash.charCodeAt(30) + 138 +
    hash.charCodeAt(22) + 86 +
    hash.charCodeAt(10) - 85 +
    hash.charCodeAt(26) + 86 +
    hash.charCodeAt(23) + 89 +
    hash.charCodeAt(19) - 56 +
    hash.charCodeAt(0) + 98 +
    hash.charCodeAt(18) - 109 +
    hash.charCodeAt(27) - 108 +
    hash.charCodeAt(6) + 102 +
    hash.charCodeAt(2) - 97 +
    hash.charCodeAt(33) - 118 +
    hash.charCodeAt(18) - 133 +
    hash.charCodeAt(37) - 120 +
    hash.charCodeAt(0) - 133 +
    hash.charCodeAt(34) - 97 +
    hash.charCodeAt(23) + 107 +
    hash.charCodeAt(38) + 142 +
    hash.charCodeAt(25) + 115 +
    hash.charCodeAt(14) - 115 +
    hash.charCodeAt(23) + 107 +
    hash.charCodeAt(6) - 123
  ).toString(16);
};

This assumes the hash is SHA1. The original function uses the length of the string if you want to use that instead:

  return Math.abs(
    e.charCodeAt(2375 % e.length) - 128 +
    e.charCodeAt(763 % e.length) + 127 +
    e.charCodeAt(1107 % e.length) + 141 +
    e.charCodeAt(318 % e.length) + 82 +
    e.charCodeAt(1711 % e.length) - 126 +
    e.charCodeAt(1903 % e.length) + 93 +
    e.charCodeAt(2044 % e.length) + 102 +
    e.charCodeAt(1275 % e.length) - 108 +
    e.charCodeAt(1329 % e.length) + 71 +
    e.charCodeAt(985 % e.length) - 130 +
    e.charCodeAt(190 % e.length) + 138 +
    e.charCodeAt(1502 % e.length) + 86 +
    e.charCodeAt(1970 % e.length) - 85 +
    e.charCodeAt(266 % e.length) + 86 +
    e.charCodeAt(2903 % e.length) + 89 +
    e.charCodeAt(579 % e.length) - 56 +
    e.charCodeAt(1600 % e.length) + 98 +
    e.charCodeAt(2458 % e.length) - 109 +
    e.charCodeAt(2547 % e.length) - 108 +
    e.charCodeAt(526 % e.length) + 102 +
    e.charCodeAt(82 % e.length) - 97 +
    e.charCodeAt(2153 % e.length) - 118 +
    e.charCodeAt(2738 % e.length) - 133 +
    e.charCodeAt(2837 % e.length) - 120 +
    e.charCodeAt(880 % e.length) - 133 +
    e.charCodeAt(1194 % e.length) - 97 +
    e.charCodeAt(663 % e.length) + 107 +
    e.charCodeAt(438 % e.length) + 142 +
    e.charCodeAt(1825 % e.length) + 115 +
    e.charCodeAt(2254 % e.length) - 115 +
    e.charCodeAt(2623 % e.length) + 107 +
    e.charCodeAt(1446 % e.length) - 123
  ).toString(16);

Also since I didn't see anyone else correct it, when you're building the SHA hash someone had mentioned:

"0" I don't know what this is, but it's part of a "or" statement and it usually defaults to this.

This is the user's id so it would only be 0 when they're not logged in.

kotori2 commented 3 years ago

@hippothon Confirmed works for me. Gonna create a PR soon. image image

cariah commented 3 years ago

Ported to python, seems to be working @hippothon

thenightex commented 3 years ago

Can confirm too. Works for me in python.

oftrash commented 3 years ago

@hippothon just emailed you.