Closed LP560 closed 3 years ago
Yeah, scrapper has started to fail Auth Attempt, went to recheck the cookie information to see if everything was in order, and couldn't find auth_hash either. Not sure if anything else changed as well.
Getting failed auth attempts here too. Using chrome, I still see auth hash but mine has changed along with sess. Updated everything and still getting failed auths.
I still see the auth_hash cookie, but I think it's because it's old. When I open an icognito window, I dont' see it anymore. However I see a cookie named "sc_is_visitor_unique" that I do not see in a normal window...
I believe Onlyfans is adding two header values using XHR called "sign" and "time". If the request header doesn't contain these two you will get an error. It isn't a cookie error. They're both generated using javascript too.
I've been seeing this issue as well. As mentioned above one of the cookies is missing on new sessions but older ones still use it.
They implemented this https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
They implemented this https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
Is that a bad thing?
Depends, but we have to reverse engineer the hash methods. I'm somewhat making progress, but I don't know if it will work.
Depends, but we have to reverse engineer the encryption methods. I'm somewhat making progress, but I don't know if it will work.
Damn, good luck bro, you the goat fr fr
@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.
@jumoog have you tried sending a request with the signing and time?
I tried sending a request with sign
and time
(and the usual stuff) and it seems to work for that specific endpoint, even multiple times, for quite a long time. I'm gonna do some more tests this weekend.
Since it's a SPA and I see a CSRF in the cookies, the AWS keys to generate the signature are probably sloppily obfuscated in vendor.js.
@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.
lol... It's not like the code source isn't available anyway 😉. And they'll known it was "figured out" if the project is still active.
@jumoog, yee it is. I didn't want to say anything yet because I didn't want them changing anything lmao. They're probably watching this thread.
lol... It's not like the code source isn't available anyway 😉. And they'll known it was "figured out" if the project is still active.
Yeah, I know, but I'd hate for them to change something whilst we're trying to figure it out.
Yay
Alright so basically...
No 2FA = no auth_hash 2FA = auth_hash and auth_uniqu
You don't need the CSRF from what I can see. Sess still needs to be passed.
Regarding the new auth headers You can pass any string for "sign" You can pass any int for "time"
As @jumoog said (which was deleted) the signing is made up of the following things
sess_token epoch timestamp URL path (queries included) user_agent "onlyfans"
an69hkjp7kj169o691ugaiaqpj
3496797795845
/api2/v2/init?app-token=33d57ade8c02dbc5a333db99ff9ae26a
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
onlyfans
You need to encode those values into sha1 encryption https://emn178.github.io/online-tools/sha1.html
I'll make the script recreate the correct signing token since OF will know if you're using an invalid token. They're probably allowing requests with the invalid sign and time values as a way to track users lmao, so I'll try my best to make it as anon as possible...
def create_sign(session, link, sess, user_agent, text="onlyfans"):
time = str(int(round(time2.time() * 1000-300000)))
path = urlparse(link).path
query = urlparse(link).query
path = path+"?"+query
a = [sess, time, path, user_agent, text]
msg = "\n".join(a)
message = msg.encode("utf-8")
hash_object = hashlib.sha1(message)
sha_1 = hash_object.hexdigest()
session.headers["sign"] = sha_1
session.headers["time"] = time
return session
Created this function if anyone else wants to use it. It'll create the signed token and add it to the session's header. Luckily the script only uses one session so I only have to call this function once.
@DIGITALCRIMINAL i think there is a bug in the most recent push to master. I'm getting
Type: Stories
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
Type: Highlights
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
Type: Posts
Scraping [photo]. Should take less than a minute.
Scraping [video | stream | gif]. Should take less than a minute.
2020-07-27 02:37:26,573 ERROR errors invalid literal for int() with base 10: ''
Traceback (most recent call last):
File "Z:\github-folders\OnlyFans\datascraper\main_datascraper.py", line 221, in start_datascraper
result = x.start_datascraper(
File "Z:\github-folders\OnlyFans\modules\onlyfans.py", line 92, in start_datascraper
results = prepare_scraper(
File "Z:\github-folders\OnlyFans\modules\onlyfans.py", line 519, in prepare_scraper
export_archive(metadata_set, archive_directory, json_settings)
File "Z:\github-folders\OnlyFans\helpers\main_helper.py", line 104, in export_archive
datas2 = ofrenamer.start(archive_path, json_settings)
File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 90, in start
metadata.valid = fix_metadata(
File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 50, in fix_metadata
reformat = prepare_reformat(model2)
File "Z:\github-folders\OnlyFans\extras\OFRenamer\start.py", line 47, in __init__
self.maximum_length = int(text_length)
ValueError: invalid literal for int() with base 10: ''
a new issue, please...
@DIGITALCRIMINAL sure i can do that, i mentioned it here since it seems like it was a side effect of this fix.
Should the auth_hash in the config be left blank with this "new" method? I don't know if I am doing something wrong, but I am still unable to log in with the script. I am doing everything else exactly the same way as before.
Edit: I tried both with and without 2FA, it tries 10 times for both V1 and V2 and then fails.
Same issue as atliax. There's no auth hash in the cookies and I'm not sure what else needs to be filled in. Is 2FA mandatory now?
If you see auth_hash in the cookies list, then yeah you gotta put it in the config
I don't see auth_hash anymore. Now there's CSRF and sc_is_visitor_unique.
Everything is the same as last time. Make sure you're using the correct user agent
My cookies are: sess, csrf, ref_src=; fp, auth_id, sc_is_visitor_unique In my config I filled in: auth_id, sess, fp and user_agent
Edit: After some digging around and testing, I am able to get past the "Auth Attempt" stage when I add the "access-token" header to the request. The next request (subscriptions) fails however, with an error code 401, "Invalid request sign".
I modified the code, so that a new "sign" is generated for each request and it started working again.
Yeah I'm still having the same issue, filled it out correctly (auth_id, authhash, sess, user_agent) and still having the Auth Attempt error
EDIT: Restarted my machine and it's working just as before. Cheers on the quick fix guys
I pulled down the new commit and it is working here. Copied over my old config.json too. Field populated are auth_id, auth_hash, sess, fp, and user_agent to match. I also explicitly set support_2fa to false if it makes a difference at all since I don't have that enabled and therefore have no authuniq field to provide.
I see auth_hash . And not CSRF and sc_is_visitor_unique. what do i have to do to CSRF and sc_is_visitor_unique. Should be visible
I couldn't get it working at first when I tried to add updated values to my configuration file, even after pulling from master, but it did work after I cleared my cookies in my browser and logged back into OnlyFans. Here's what I did practically:
onlyfans.com
(using developer tools).
Application
tabhttps://onlyfans.com/
and Clear
(Screenshot)Storage
tab, instead of application tab.Delete All from ".onlyfans.com"
. Screenshotauth_uniq_XXXXXXX
- Looks like it only shows up if you have 2FA enabled and you check the "Save this computer for 30 days" option.auth_id
- Unchangedsess
fp
From what I can tell, csrf
or any other value in cookies are not necessary. At least not for the time being.
I no longer had the auth_hash
cookie value at all, so I just entered that as a blank string in the JSON config file - See short example:
...
"auth_id": "XXXXX",
"auth_hash": "",
"auth_uniq_": "YYYYYYYYYYYYYY",
...
I no longer had the
auth_hash
cookie value at all
Same here, and it is working.
Interesting... I tried the latest commit again, doing everything exactly the same as yesterday and now it works perfectly.
The only difference I can think of is that yesterday the old version had been running recently, hammering the OF servers with "invalid sign" requests for a few hours. Maybe that's what caused my problems with the new version.
OnlyFans stopped checking signed requests... I'm still going to keep it there though.
Is it down again? Just tried it and getting the same error. Didn't change anything, tried restarting my machine, clearing cookies and still the same issue
It's been blocked again. Time for another sleepless night.
Damn, take your time broski, inspect element ftw XD
It works fine for me? Or did they unblock it again?
@Hashirama @Anonymeowse fixed it in the latest commit
To me the latest commit seems to authenticate (it shows my name) but then I get this error: 2020-07-28 15:50:04,143 ERROR errors 'access-token' Traceback (most recent call last): File "d:\OneDrive\cb\grab_of5\datascraper\main_datascraper.py", line 120, in start_datascraper session["session"], app_token, session["subscriber_count"], me_api, auth_count) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 783, in get_subscriptions offset_array, [session])) File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 276, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 657, in get raise self._value File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\multiprocessing\pool.py", line 47, in starmapstar return list(itertools.starmap(args[0], args[1])) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 769, in multi r = json_request(session, link) File "d:\OneDrive\cb\grab_of5\helpers\main_helper.py", line 219, in json_request if session.headers["access-token"]: File "D:\mrs\WPy64-3740\python-3.7.4.amd64\lib\site-packages\requests\structures.py", line 52, in getitem return self._store[key.lower()][1] KeyError: 'access-token'
I'a a creator so sometimes I have different errors. I'm also not using 2FA (don't know if useul info). I'm using same conf.json as before except I removed auth_hash
Ye, it's a performer issue. It's because the script creates an entirely new request.Session() when getting subscribers as a performer.
@mwald84 I think the latest commit fixes it. It should include your username and others when getting subscribers. If it's not fixed, I'll just login to my creator account.
Using the last commit:
2020-07-28 16:20:53,439 ERROR errors 'subscribedBy' Traceback (most recent call last): File "d:\OneDrive\cb\grab_of5\datascraper\main_datascraper.py", line 120, in start_datascraper session["session"], app_token, session["subscriber_count"], me_api, auth_count) File "d:\OneDrive\cb\grab_of5\modules\onlyfans.py", line 811, in get_subscriptions subscribedBy = result["subscribedBy"] KeyError: 'subscribedBy'
Should work now with the latest commit. I also had to update the create sign function.
It works! Thanks!
I know this is not the proper place for this (please let me know if there's a forum for questions) but do you think they check IP during authentication? I made a simple script to only authenticate, check the number of posts from a speciic model and print on screen. I used to run this on a remote server. Now, with the new authentication, I can run the script on my computer (same I used the browser) but it can't autenticate on the remote server. Other than IP, the only diffference I can think of is that it runs on Linux (and my computer is Win10)
i've been trying to get it to work for ages now and despite filling in all the required information, i either get: ....... Auth Attempt 10/10 Access denied. 2020-07-29 01:01:26,272 ERROR errors 'app_token' Traceback (most recent call last): File "J:\Python\OnlyFans-5.1\datascraper\main_datascraper.py", line 92, in sta rt_datascraper app_token = json_auth['app_token'] KeyError: 'app_token'
or i get nothing but failed attempts and the message "There's nothing to scrape."
am using auth_id, auth_hash, sess, fp, user_agent but it doesn't matter if any are left out as the result is the same.
@dannylaroos Is app_token
set in your config.json alongside the fields you listed in your comment?
I couldn't get it working at first when I tried to add updated values to my configuration file, even after pulling from master, but it did work after I cleared my cookies in my browser and logged back into OnlyFans. Here's what I did practically:
Cleared cookies for
onlyfans.com
(using developer tools).
In Brave (and Chrome probably):
- Open OnlyFans.com in my browser
- Hit F12 or CTRL+Shift+I for Developer Tools
Application
tab- Cookies
- Right-click on
https://onlyfans.com/
andClear
(Screenshot)Firefox Developer Tools are pretty similar, except:
- Step 3:
Storage
tab, instead of application tab.- Step 5: Right-click on the cookie values instead (not the URL) and click
Delete All from ".onlyfans.com"
. Screenshot- Of course, an alternative is to just clear all of your cookies, but it might be a bit overkill :)
- Log back in as normal
Fill in new cookie values, where I had the following:
auth_uniq_XXXXXXX
- Looks like it only shows up if you have 2FA enabled and you check the "Save this computer for 30 days" option.auth_id
- Unchangedsess
fp
From what I can tell,
csrf
or any other value in cookies are not necessary. At least not for the time being.I no longer had the
auth_hash
cookie value at all, so I just entered that as a blank string in the JSON config file - See short example:... "auth_id": "XXXXX", "auth_hash": "", "auth_uniq_": "YYYYYYYYYYYYYY", ...
This worked for me
@dannylaroos Is
app_token
set in your config.json alongside the fields you listed in your comment?
yes it is. forgot to mention it first time round.
gonna keep experimenting.
edit: tried some of the above methods without success but gave it one last go just now and all of a sudden it works!
only needed the "auth_id", "sess" "app-token" and "user_agent"
~hmm my stuff just randomly stopped working again.~
nevermind i just had to update my config.json again, which is weird since normally it takes weeks in-between times i need to update the sess/fp/etc. I wonder they made the time to live shorter.
it seems OF has removed auth_hash when you inspect. is there any other way to find it? Or login