DIGITALCRIMINAL / ArchivedUltimaScraper

Scrape content from OnlyFans and Fansly
GNU General Public License v3.0
953 stars 39 forks source link

Getting KeyError: 'auth' #622

Closed sati-space closed 1 year ago

sati-space commented 2 years ago

I already filled in the auth.json file but I keep getting the same error to scrape OF data

Choose Sites: 0 = All | 1 = OnlyFans | 2 = Fansly | 3 = StarsAVN
1
Traceback (most recent call last):
  File "/Users/ricardodelgado/source/OnlyFans/start_ofd.py", line 66, in <module>
    asyncio.run(main())
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Users/ricardodelgado/source/OnlyFans/start_ofd.py", line 52, in main
    api = await main_datascraper.start_datascraper(config, site_name)
  File "/Users/ricardodelgado/source/OnlyFans/datascraper/main_datascraper.py", line 131, in start_datascraper
    await default(datascraper)
  File "/Users/ricardodelgado/source/OnlyFans/datascraper/main_datascraper.py", line 55, in default
    await main_helper.process_profiles(api, global_settings)
  File "/Users/ricardodelgado/source/OnlyFans/helpers/main_helper.py", line 963, in process_profiles
    json_auth = temp_json_auth["auth"]
KeyError: 'auth'

Anyone found a fix?

GooseWhoIsLoose commented 2 years ago

Yeah, Onlyfans changed some stuff. In #624 I linked some stuff that'll help, particularly changing line 270 of create_user.py and/or line 283 of user_model.py to results = [create_highlight(x) for x in results.get("list")]

Let me know if you need any more help :)

sati-space commented 2 years ago

@GooseWhoIsLoose Thank you for your help. I changed line 270 from create_user.py and 283 from user_model.py to the one you posted above. I also changed the config.json to a comment I saw in #604 and start_ofl.py I saw in #496 but I still have no luck. Still having the same KeyError: 'auth' error

GooseWhoIsLoose commented 2 years ago

Hmmm....

Man, I hate programs that don't have GUIs (and good ones, at that). I get how much easier this is from a developer standpoint, but a functional GUI that can direct you where you need to go is so incredibly helpful. Doesn't need to be pretty, just helpful.

Oh well. Still very grateful to have this program, it works better than anything else I've found.

Read through this. I had trouble getting this program working when I first downloaded it, but after following this PDF to the letter (including installing Python direct from the Microsoft Store), I got it to work. I don't remember exactly which errors it fixed, but it's worth a look. This YouTube video was helpful as well.

Good luck! I hope this stuff helps you! :)

https://www.youtube.com/watch?v=uP4N2Ftg5us

howtousereadme.pdf

Tramp-ish commented 2 years ago

@GooseWhoIsLoose Thank you for your help. I changed line 270 from create_user.py and 283 from user_model.py to the one you posted above. I also changed the config.json to a comment I saw in #604 and start_ofl.py I saw in #496 but I still have no luck. Still having the same KeyError: 'auth' error

I believe I have solved part of the issue: In addition to the changes suggested in the comments above, change lines 963-964 in main_helper.py to say: json_auth = temp_json_auth.get("auth") if (json_auth is None) or (json_auth.get("active") is None):

However even after fixing this, I still cant download anything despite being able to log in. Edit: If this was never an issue for you, you can ignore this: I managed to fix the issue above by setting false the messages and paid_content scrape jobs in config.json It's not an ideal fix, but at least you can download vids and pics from the main page now.

sati-space commented 2 years ago

@GooseWhoIsLoose Thank you for your help. I changed line 270 from create_user.py and 283 from user_model.py to the one you posted above. I also changed the config.json to a comment I saw in #604 and start_ofl.py I saw in #496 but I still have no luck. Still having the same KeyError: 'auth' error

I believe I have solved part of the issue: In addition to the changes suggested in the comments above, change lines 963-964 in main_helper.py to say: json_auth = temp_json_auth.get("auth") if (json_auth is None) or (json_auth.get("active") is None):

However even after fixing this, I still cant download anything despite being able to log in. Edit: If this was never an issue for you, you can ignore this: I managed to fix the issue above by setting false the messages and paid_content scrape jobs in config.json It's not an ideal fix, but at least you can download vids and pics from the main page now.

@Tramp-ish The KeyError: 'auth' error was indeed fixed like you said. What I can see is that it's only scraping main page, and when I tried selecting only one model, it scraped whatever I had historically from previous subscriptions that were posted on the main page. I did left the 'messages' and 'paid_content' to true to see what was the output, and that was the result. Still not able to access the private areas of the paid_content.

GeekProgram007 commented 2 years ago

@GooseWhoIsLoose Thank you for your help. I changed line 270 from create_user.py and 283 from user_model.py to the one you posted above. I also changed the config.json to a comment I saw in #604 and start_ofl.py I saw in #496 but I still have no luck. Still having the same KeyError: 'auth' error

I believe I have solved part of the issue: In addition to the changes suggested in the comments above, change lines 963-964 in main_helper.py to say: json_auth = temp_json_auth.get("auth") if (json_auth is None) or (json_auth.get("active") is None):

However even after fixing this, I still cant download anything despite being able to log in. Edit: If this was never an issue for you, you can ignore this: I managed to fix the issue above by setting false the messages and paid_content scrape jobs in config.json It's not an ideal fix, but at least you can download vids and pics from the main page now.

this fixed the issue! I can scrape with no issues. On the latest commit as of 10 Sept 2022 and changing this line. The old code was

if (json_auth is None) or (json_auth.get("active") is None) is the new line and much more "Python-ic."

Old line was if not json_auth.get("active", None): which is never seen in Python tutorials lol.

Many thanks for the fix! Great mini puzzle to solve. I enabled messages / private data and it works.