UltimaHoarder / UltimaScraper

Scrape all the media from an OnlyFans account - Updated regularly
GNU General Public License v3.0
3.99k stars 609 forks source link

Auth Failures (echo chamber) #341

Closed LP560 closed 3 years ago

LP560 commented 4 years ago

it seems OF has removed auth_hash when you inspect. is there any other way to find it? Or login

WellSomething commented 4 years ago

Yeah, scrapper has started to fail Auth Attempt, went to recheck the cookie information to see if everything was in order, and couldn't find auth_hash either. Not sure if anything else changed as well.

artyom2035 commented 4 years ago

Getting failed auth attempts here too. Using chrome, I still see auth hash but mine has changed along with sess. Updated everything and still getting failed auths.

jeremfg commented 4 years ago

I still see the auth_hash cookie, but I think it's because it's old. When I open an icognito window, I dont' see it anymore. However I see a cookie named "sc_is_visitor_unique" that I do not see in a normal window...

image

Hashirama commented 4 years ago

I believe Onlyfans is adding two header values using XHR called "sign" and "time". If the request header doesn't contain these two you will get an error. It isn't a cookie error. They're both generated using javascript too.

shinji257 commented 4 years ago

I've been seeing this issue as well. As mentioned above one of the cookies is missing on new sessions but older ones still use it.

UltimaHoarder commented 4 years ago

They implemented this https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

BoxFit3 commented 4 years ago

They implemented this https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Is that a bad thing?

UltimaHoarder commented 4 years ago

Depends, but we have to reverse engineer the hash methods. I'm somewhat making progress, but I don't know if it will work.

BoxFit3 commented 4 years ago

Depends, but we have to reverse engineer the encryption methods. I'm somewhat making progress, but I don't know if it will work.

Damn, good luck bro, you the goat fr fr

UltimaHoarder commented 4 years ago

@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.

UltimaHoarder commented 4 years ago

@jumoog have you tried sending a request with the signing and time?

mileyvirus commented 4 years ago

I tried sending a request with sign and time (and the usual stuff) and it seems to work for that specific endpoint, even multiple times, for quite a long time. I'm gonna do some more tests this weekend.

mileyvirus commented 4 years ago

Since it's a SPA and I see a CSRF in the cookies, the AWS keys to generate the signature are probably sloppily obfuscated in vendor.js.

jeremfg commented 4 years ago

@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.

lol... It's not like the code source isn't available anyway 😉. And they'll known it was "figured out" if the project is still active.

UltimaHoarder commented 4 years ago

@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.

lol... It's not like the code source isn't available anyway 😉. And they'll known it was "figured out" if the project is still active.

Yeah, I know, but I'd hate for them to change something whilst we're trying to figure it out.

UltimaHoarder commented 4 years ago

Yay image

UltimaHoarder commented 4 years ago

Alright so basically...

No 2FA = no auth_hash 2FA = auth_hash and auth_uniqu

You don't need the CSRF from what I can see. Sess still needs to be passed.

Regarding the new auth headers You can pass any string for "sign" You can pass any int for "time"

As @jumoog said (which was deleted) the signing is made up of the following things

sess_token epoch timestamp URL path (queries included) user_agent "onlyfans"

an69hkjp7kj169o691ugaiaqpj
3496797795845
/api2/v2/init?app-token=33d57ade8c02dbc5a333db99ff9ae26a
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
onlyfans

You need to encode those values into sha1 encryption https://emn178.github.io/online-tools/sha1.html

I'll make the script recreate the correct signing token since OF will know if you're using an invalid token. They're probably allowing requests with the invalid sign and time values as a way to track users lmao, so I'll try my best to make it as anon as possible...

UltimaHoarder commented 4 years ago
def create_sign(session, link, sess, user_agent, text="onlyfans"):
    time = str(int(round(time2.time() * 1000-300000)))
    path = urlparse(link).path
    query = urlparse(link).query
    path = path+"?"+query
    a = [sess, time, path, user_agent, text]
    msg = "\n".join(a)
    message = msg.encode("utf-8")
    hash_object = hashlib.sha1(message)
    sha_1 = hash_object.hexdigest()
    session.headers["sign"] = sha_1
    session.headers["time"] = time
    return session

Created this function if anyone else wants to use it. It'll create the signed token and add it to the session's header. Luckily the script only uses one session so I only have to call this function once.

ghost commented 4 years ago

@DIGITALCRIMINAL i think there is a bug in the most recent push to master. I'm getting

Type: Stories
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
Type: Highlights
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
Type: Posts
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
2020-07-27 02:37:26,573 ERROR errors invalid literal for int() with base 10: ''
Traceback (most recent call last):
  File "Z:\github-folders\OnlyFans\datascraper\main_datascraper.py", line 221, in start_datascraper
    result = x.start_datascraper(
  File "Z:\github-folders\OnlyFans\modules\onlyfans.py", line 92, in start_datascraper
    results = prepare_scraper(
  File "Z:\github-folders\OnlyFans\modules\onlyfans.py", line 519, in prepare_scraper
    export_archive(metadata_set, archive_directory, json_settings)
  File "Z:\github-folders\OnlyFans\helpers\main_helper.py", line 104, in export_archive
    datas2 = ofrenamer.start(archive_path, json_settings)
  File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 90, in start
    metadata.valid = fix_metadata(
  File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 50, in fix_metadata
    reformat = prepare_reformat(model2)
  File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 47, in __init__
    self.maximum_length = int(text_length)
ValueError: invalid literal for int() with base 10: ''
UltimaHoarder commented 4 years ago

a new issue, please...

ghost commented 4 years ago

@DIGITALCRIMINAL sure i can do that, i mentioned it here since it seems like it was a side effect of this fix.

atliax commented 4 years ago

Should the auth_hash in the config be left blank with this "new" method? I don't know if I am doing something wrong, but I am still unable to log in with the script. I am doing everything else exactly the same way as before.

Edit: I tried both with and without 2FA, it tries 10 times for both V1 and V2 and then fails.

NotMyMainOne commented 4 years ago

Same issue as atliax. There's no auth hash in the cookies and I'm not sure what else needs to be filled in. Is 2FA mandatory now?

UltimaHoarder commented 4 years ago

If you see auth_hash in the cookies list, then yeah you gotta put it in the config

NotMyMainOne commented 4 years ago

I don't see auth_hash anymore. Now there's CSRF and sc_is_visitor_unique.

UltimaHoarder commented 4 years ago

Everything is the same as last time. Make sure you're using the correct user agent

atliax commented 4 years ago

My cookies are: sess, csrf, ref_src=; fp, auth_id, sc_is_visitor_unique In my config I filled in: auth_id, sess, fp and user_agent

Edit: After some digging around and testing, I am able to get past the "Auth Attempt" stage when I add the "access-token" header to the request. The next request (subscriptions) fails however, with an error code 401, "Invalid request sign".

I modified the code, so that a new "sign" is generated for each request and it started working again.

BoxFit3 commented 4 years ago

Yeah I'm still having the same issue, filled it out correctly (auth_id, authhash, sess, user_agent) and still having the Auth Attempt error

EDIT: Restarted my machine and it's working just as before. Cheers on the quick fix guys

shinji257 commented 4 years ago

I pulled down the new commit and it is working here. Copied over my old config.json too. Field populated are auth_id, auth_hash, sess, fp, and user_agent to match. I also explicitly set support_2fa to false if it makes a difference at all since I don't have that enabled and therefore have no authuniq field to provide.

image

Dalla-maker commented 4 years ago

I see auth_hash . And not CSRF and sc_is_visitor_unique. what do i have to do to CSRF and sc_is_visitor_unique. Should be visible

AlexDucks commented 4 years ago

I couldn't get it working at first when I tried to add updated values to my configuration file, even after pulling from master, but it did work after I cleared my cookies in my browser and logged back into OnlyFans. Here's what I did practically:

  1. Cleared cookies for onlyfans.com (using developer tools).
    • In Brave (and Chrome probably):
      1. Open OnlyFans.com in my browser
      2. Hit F12 or CTRL+Shift+I for Developer Tools
      3. Application tab
      4. Cookies
      5. Right-click on https://onlyfans.com/ and Clear (Screenshot)
    • Firefox Developer Tools are pretty similar, except:
      • Step 3: Storage tab, instead of application tab.
      • Step 5: Right-click on the cookie values instead (not the URL) and click Delete All from ".onlyfans.com". Screenshot
    • Of course, an alternative is to just clear all of your cookies, but it might be a bit overkill :)
  2. Log back in as normal
  3. Fill in new cookie values, where I had the following:
    • auth_uniq_XXXXXXX - Looks like it only shows up if you have 2FA enabled and you check the "Save this computer for 30 days" option.
    • auth_id - Unchanged
    • sess
    • fp

From what I can tell, csrf or any other value in cookies are not necessary. At least not for the time being.

I no longer had the auth_hash cookie value at all, so I just entered that as a blank string in the JSON config file - See short example:

...
"auth_id": "XXXXX",
"auth_hash": "",
"auth_uniq_": "YYYYYYYYYYYYYY",
...
BatmanAndWilly commented 4 years ago

I no longer had the auth_hash cookie value at all

Same here, and it is working.

atliax commented 4 years ago

Interesting... I tried the latest commit again, doing everything exactly the same as yesterday and now it works perfectly.

The only difference I can think of is that yesterday the old version had been running recently, hammering the OF servers with "invalid sign" requests for a few hours. Maybe that's what caused my problems with the new version.

UltimaHoarder commented 4 years ago

OnlyFans stopped checking signed requests... I'm still going to keep it there though.

BoxFit3 commented 4 years ago

Is it down again? Just tried it and getting the same error. Didn't change anything, tried restarting my machine, clearing cookies and still the same issue

UltimaHoarder commented 4 years ago

It's been blocked again. Time for another sleepless night. image

BoxFit3 commented 4 years ago

Damn, take your time broski, inspect element ftw XD

Hashirama commented 4 years ago

It works fine for me? Or did they unblock it again?

UltimaHoarder commented 4 years ago

@Hashirama @Anonymeowse fixed it in the latest commit

mwald84 commented 4 years ago

To me the latest commit seems to authenticate (it shows my name) but then I get this error: 2020-07-28 15:50:04,143 ERROR errors 'access-token' Traceback (most recent call last): File "d:\OneDrive\cb\grab_of5\datascraper\main_datascraper.py", line 120, in start_datascraper session["session"], app_token, session["subscriber_count"], me_api, auth_count) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 783, in get_subscriptions offset_array, [session])) File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 276, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 657, in get raise self._value File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 47, in starmapstar return list(itertools.starmap(args[0], args[1])) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 769, in multi r = json_request(session, link) File "d:\OneDrive\cb\grab_of5\helpers\main_helper.py", line 219, in json_request if session.headers["access-token"]: File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\site-packages\requests\structures.py", line 52, in getitem return self._store[key.lower()][1] KeyError: 'access-token'

I'a a creator so sometimes I have different errors. I'm also not using 2FA (don't know if useul info). I'm using same conf.json as before except I removed auth_hash

UltimaHoarder commented 4 years ago

Ye, it's a performer issue. It's because the script creates an entirely new request.Session() when getting subscribers as a performer.

@mwald84 I think the latest commit fixes it. It should include your username and others when getting subscribers. If it's not fixed, I'll just login to my creator account.

mwald84 commented 4 years ago

Using the last commit:

2020-07-28 16:20:53,439 ERROR errors 'subscribedBy' Traceback (most recent call last): File "d:\OneDrive\cb\grab_of5\datascraper\main_datascraper.py", line 120, in start_datascraper session["session"], app_token, session["subscriber_count"], me_api, auth_count) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 811, in get_subscriptions subscribedBy = result["subscribedBy"] KeyError: 'subscribedBy'

UltimaHoarder commented 4 years ago

Should work now with the latest commit. I also had to update the create sign function.

mwald84 commented 4 years ago

It works! Thanks!

mwald84 commented 4 years ago

I know this is not the proper place for this (please let me know if there's a forum for questions) but do you think they check IP during authentication? I made a simple script to only authenticate, check the number of posts from a speciic model and print on screen. I used to run this on a remote server. Now, with the new authentication, I can run the script on my computer (same I used the browser) but it can't autenticate on the remote server. Other than IP, the only diffference I can think of is that it runs on Linux (and my computer is Win10)

dannylaroos commented 4 years ago

i've been trying to get it to work for ages now and despite filling in all the required information, i either get: ....... Auth Attempt 10/10 Access denied. 2020-07-29 01:01:26,272 ERROR errors 'app_token' Traceback (most recent call last): File "J:\Python\OnlyFans-5.1\datascraper\main_datascraper.py", line 92, in sta rt_datascraper app_token = json_auth['app_token'] KeyError: 'app_token'

or i get nothing but failed attempts and the message "There's nothing to scrape."

am using auth_id, auth_hash, sess, fp, user_agent but it doesn't matter if any are left out as the result is the same.

Anonymeowse commented 4 years ago

@dannylaroos Is app_token set in your config.json alongside the fields you listed in your comment?

zedawing commented 4 years ago

I couldn't get it working at first when I tried to add updated values to my configuration file, even after pulling from master, but it did work after I cleared my cookies in my browser and logged back into OnlyFans. Here's what I did practically:

  1. Cleared cookies for onlyfans.com (using developer tools).

    • In Brave (and Chrome probably):

      1. Open OnlyFans.com in my browser
      2. Hit F12 or CTRL+Shift+I for Developer Tools
      3. Application tab
      4. Cookies
      5. Right-click on https://onlyfans.com/ and Clear (Screenshot)
    • Firefox Developer Tools are pretty similar, except:

      • Step 3: Storage tab, instead of application tab.
      • Step 5: Right-click on the cookie values instead (not the URL) and click Delete All from ".onlyfans.com". Screenshot
    • Of course, an alternative is to just clear all of your cookies, but it might be a bit overkill :)
  2. Log back in as normal
  3. Fill in new cookie values, where I had the following:

    • auth_uniq_XXXXXXX - Looks like it only shows up if you have 2FA enabled and you check the "Save this computer for 30 days" option.
    • auth_id - Unchanged
    • sess
    • fp

From what I can tell, csrf or any other value in cookies are not necessary. At least not for the time being.

I no longer had the auth_hash cookie value at all, so I just entered that as a blank string in the JSON config file - See short example:

...
"auth_id": "XXXXX",
"auth_hash": "",
"auth_uniq_": "YYYYYYYYYYYYYY",
...

This worked for me

dannylaroos commented 4 years ago

@dannylaroos Is app_token set in your config.json alongside the fields you listed in your comment?

yes it is. forgot to mention it first time round.

gonna keep experimenting.

edit: tried some of the above methods without success but gave it one last go just now and all of a sudden it works!

only needed the "auth_id", "sess" "app-token" and "user_agent"

ghost commented 4 years ago

~hmm my stuff just randomly stopped working again.~

nevermind i just had to update my config.json again, which is weird since normally it takes weeks in-between times i need to update the sess/fp/etc. I wonder they made the time to live shorter.