DIGITALCRIMINAL / ArchivedUltimaScraper

Scrape content from OnlyFans and Fansly
GNU General Public License v3.0
948 stars 39 forks source link

Traceback Error: AttributeError: 'str' object has no attribute 'get' #518

Closed Nostang3 closed 1 year ago

Nostang3 commented 2 years ago

Just started this morning. Tried updating all modules in requirements. Running 7.6.1. Tried changing the auto_api_choice to "2,3,4,5,6,7" that I saw in another thread with no change.

0
Scrape Processing
Name: enjinight
Type: Profile
Type: Stories
Traceback (most recent call last):
  File "D:\OnlyFans\start_ofd.py", line 104, in <module>
    asyncio.run(main())
  File "C:\Users\chris\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\chris\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
    return future.result()
  File "D:\OnlyFans\start_ofd.py", line 86, in main
    api = await main_datascraper.start_datascraper(
  File "D:\OnlyFans\datascraper\main_datascraper.py", line 94, in start_datascraper
    await main_helper.process_names(
  File "D:\OnlyFans\helpers\main_helper.py", line 966, in process_names
    result = await module.start_datascraper(authed, username, site_name)
  File "D:\OnlyFans\modules\onlyfans.py", line 161, in start_datascraper
    await prepare_scraper(authed, site_name, item)
  File "D:\OnlyFans\modules\onlyfans.py", line 784, in prepare_scraper
    highlights = await subscription.get_highlights()
  File "D:\OnlyFans\apis\onlyfans\classes\create_user.py", line 270, in get_highlights
    results = [create_highlight(x) for x in results]
  File "D:\OnlyFans\apis\onlyfans\classes\create_user.py", line 270, in <listcomp>
    results = [create_highlight(x) for x in results]
  File "D:\OnlyFans\apis\onlyfans\classes\create_highlight.py", line 3, in __init__
    self.id: int = option.get("id")
AttributeError: 'str' object has no attribute 'get'
lordoffools commented 2 years ago

You need to have the module installed: pip install yarl

i did that and it shows me some other module not found, i do pip install for that and still everytime it keeps showing some or other module missing

Keep going. You'll eventually get them all.

cornerdebug commented 2 years ago

after doing this , when i enter the profile in scraping, it says finished scraping but doesnt scrape a single content

Did you manage to get that resolved?

gmmathews23 commented 2 years ago

after doing this , when i enter the profile in scraping, it says finished scraping but doesnt scrape a single content

I have this issue too, it appears to be only scraping messages no matter what I select

TheMissingPort commented 2 years ago

In the file apis\onlyfans\classes\create_user.py change the line 270 from: results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")]

Don't need to change your auto_media_choice or auto_api_choice

What if I dont have a "create_user.py" file in that folder?

Smackdab69 commented 2 years ago

I am running on the latest build and made the suggested change and it still doesn't work :( @DIGITALCRIMINALS

In the file apis\onlyfans\classes\create_user.py change the line 270 from: results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")] Don't need to change your auto_media_choice or auto_api_choice

Did you save the file? Did you verify later that it was saved? Are you getting the exact same error as above?

Yes, I saved the file (made the change in _usermodel.py. The error I get now is below:

Scrape Processing Name: daisykeech Type: Profile Type: Stories Scrape Attempt: 1/100 Traceback (most recent call last): File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\start_ofd.py", line 77, in asyncio.run(main()) File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete return future.result() File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\start_ofd.py", line 62, in main api = await main_datascraper.start_datascraper( File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\datascraper\main_datascraper.py", line 98, in start_datascraper await main_helper.process_names( File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\helpers\main_helper.py", line 1023, in process_names result = await module.start_datascraper(authed, username, site_name) File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\modules\onlyfans.py", line 164, in start_datascraper await prepare_scraper(authed, site_name, item) File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\modules\onlyfans.py", line 781, in prepare_scraper highlights = await subscription.get_highlights() File "\192.168.2.128\d\1_Ripping_OnlyFans\OnlyFans 2022 Ripping\apis\onlyfans\classes\user_model.py", line 275, in get_highlights results = [create_highlight(x) for x in results.get("data")] AttributeError: 'list' object has no attribute 'get'

Exception ignored in: <function _ProactorBasePipeTransport.del at 0x00000232F7771FC0> Traceback (most recent call last): File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\proactor_events.py", line 116, in del self.close() File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\proactor_events.py", line 108, in close self._loop.call_soon(self._call_connection_lost, None) File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 745, in call_soon self._check_closed() File "C:\Users\server\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 510, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

TheMissingPort commented 2 years ago

In the file apis\onlyfans\classes\create_user.py change the line 270 from: results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")]

Don't need to change your auto_media_choice or auto_api_choice

I also changed it in "user_model.py" because that's where my error is, not sure why

Now the error changed to :

-> AttributeError: 'list' object has no attribute 'get'

Any ideas? Thanks

thepetcow commented 2 years ago

I am still receiving the error even though I've updated line 283 to results = [create_highlight(x) for x in results["data"]]

Type: Profile
Type: Stories
Traceback (most recent call last):
  File "M:\Ripper\OnlyFans-7.6.1\start_ofd.py", line 104, in <module>
    asyncio.run(main())
  File "C:\Users\Squeaky\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Squeaky\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
    return future.result()
  File "M:\Ripper\OnlyFans-7.6.1\start_ofd.py", line 86, in main
    api = await main_datascraper.start_datascraper(
  File "M:\Ripper\OnlyFans-7.6.1\datascraper\main_datascraper.py", line 94, in start_datascraper
    await main_helper.process_names(
  File "M:\Ripper\OnlyFans-7.6.1\helpers\main_helper.py", line 966, in process_names
    result = await module.start_datascraper(authed, username, site_name)
  File "M:\Ripper\OnlyFans-7.6.1\modules\onlyfans.py", line 161, in start_datascraper
    await prepare_scraper(authed, site_name, item)
  File "M:\Ripper\OnlyFans-7.6.1\modules\onlyfans.py", line 784, in prepare_scraper
    highlights = await subscription.get_highlights()
  File "M:\Ripper\OnlyFans-7.6.1\apis\onlyfans\classes\create_user.py", line 270, in get_highlights
    results = [create_highlight(x) for x in results]
  File "M:\Ripper\OnlyFans-7.6.1\apis\onlyfans\classes\create_user.py", line 270, in <listcomp>
    results = [create_highlight(x) for x in results]
  File "M:\Ripper\OnlyFans-7.6.1\apis\onlyfans\classes\create_highlight.py", line 3, in __init__
    self.id: int = option.get("id")
AttributeError: 'str' object has no attribute 'get'

EDIT: Needed to change it in the both files (create_user.py and user_model.py).

DawgNewb commented 2 years ago

In the file apis\onlyfans\classes\create_user.py change the line 270 from: results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")] Don't need to change your auto_media_choice or auto_api_choice

for those using latest commit apis\onlyfans\classes\user_model.py line 283 is where you need to look for this instead

This one worked for me I can confirm.

the line for me is 282 not 283 work fine did change it.. Your post did help me. Thanks

I can confirm. It's working with 7.6.1 by changing line 282 in "apis\onlyfans\classes\user_model.py" from results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")]

JamiDEV commented 2 years ago

@naodrej how do get in touch with you, I have a scraper script for coomer.party I need help with.

ghost commented 2 years ago

@cuenta111

In the file apis\onlyfans\classes\create_user.py change the line 270 from: results = [create_highlight(x) for x in results] to results = [create_highlight(x) for x in results.get("data")] Don't need to change your auto_media_choice or auto_api_choice

after doing this , when i enter the profile in scraping, it says finished scraping but doesnt scrape a single content

Same for me. No errors, just this. Same for other models. It correctly creates Avatars and Headers folders with pics inside tho

Scrape Processing
Name: roxysdreamfree
Type: Profile
Type: Stories
No Stories Found.
Type: Posts
Scrape Attempt: 1/100
Type: Archived Posts
Processing Scraped Posts
100%|███████████████████████████████████████████████████████████████████████████████| 99/99 [00:00<00:00, 99744.44it/s]
Processing metadata.
Finished processing metadata.
Renaming files.
0it [00:00, ?it/s]
Type: Archived
No Archived Found.
Type: Messages
Processing Scraped Messages
100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<?, ?it/s]
Processing metadata.
Finished processing metadata.
Renaming files.
0it [00:00, ?it/s]
Scrape Completed

Archive Completed in 0.33 Minutes
lucky1luc commented 2 years ago

File "D:\Data\OnlyFans-7.6.1\apis\onlyfans\classes\create_user.py", line 270, in get_highlights results = [create_highlight(x) for x in results.get("data")] TypeError: 'NoneType' object is not iterable

simonchin513 commented 2 years ago

File "D:\Data\OnlyFans-7.6.1\apis\onlyfans\classes\create_user.py", line 270, in get_highlights results = [create_highlight(x) for x in results.get("data")] TypeError: 'NoneType' object is not iterable

Can confirm, still getting this problem

lordoffools commented 2 years ago

Indeed, this issue has re-surfaced.

JamiDEV commented 2 years ago

@lordoffools going back to setting the api choice resolved the issue.

Nostang3 commented 2 years ago

Can confirm that the problem was fixed a while ago with the line edit but now I'm getting this error now.

Traceback (most recent call last):
  File "D:\OnlyFans\start_ofd.py", line 104, in <module>
    asyncio.run(main())
  File "C:\Users\chris\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\chris\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
    return future.result()
  File "D:\OnlyFans\start_ofd.py", line 86, in main
    api = await main_datascraper.start_datascraper(
  File "D:\OnlyFans\datascraper\main_datascraper.py", line 94, in start_datascraper
    await main_helper.process_names(
  File "D:\OnlyFans\helpers\main_helper.py", line 966, in process_names
    result = await module.start_datascraper(authed, username, site_name)
  File "D:\OnlyFans\modules\onlyfans.py", line 161, in start_datascraper
    await prepare_scraper(authed, site_name, item)
  File "D:\OnlyFans\modules\onlyfans.py", line 784, in prepare_scraper
    highlights = await subscription.get_highlights()
  File "D:\OnlyFans\apis\onlyfans\classes\create_user.py", line 270, in get_highlights
    results = [create_highlight(x) for x in results.get("data")]
TypeError: 'NoneType' object is not iterable
Nick05491 commented 2 years ago

See potential fix here: https://github.com/DIGITALCRIMINALS/OnlyFans/issues/604#issuecomment-1227612486

Nostang3 commented 2 years ago

See potential fix here: #604 (comment)

Weird but I didn't have a user_model.py in my apis/onlyfans/classes/

Adding it to the folder didn't change anything.

valdearg commented 2 years ago

See potential fix here: #604 (comment)

Weird but I didn't have a user_model.py in my apis/onlyfans/classes/

Adding it to the folder didn't change anything.

You may be running the "release" version of the ripper, if you download the full repository from the home page and select Code > Download ZIP. If you overwrite the files you have downloaded, excluding your auth file you should get the right files.

Nostang3 commented 2 years ago

You may be running the "release" version of the ripper, if you download the full repository from the home page and select Code > Download ZIP. If you overwrite the files you have downloaded, excluding your auth file you should get the right files.

You were right, I was. I have since downloaded the zip and copied over the files and replaced it with the the one you linked and I was able to scrape all of of my subscriptions. Thanks for the help.

Fall0n commented 2 years ago

I think this problem has to do with the fact that when you download, the script shuts down and you don't have time to see the error messages. Anyone know how I can fix this?

ghost commented 2 years ago

I think this problem has to do with the fact that when you download, the script shuts down and you don't have time to see the error messages. Anyone know how I can fix this?

if you want the script to display everything it is doing you could try running it via the powershell with -v attribute

Fall0n commented 2 years ago

I think this problem has to do with the fact that when you download, the script shuts down and you don't have time to see the error messages. Anyone know how I can fix this?

if you want the script to display everything it is doing you could try running it via the powershell with -v attribute

thanks for info i get this error

Scrape Attempt: 1/100 Traceback (most recent call last): File "C:\Temp\OnlyFans-master\start_ofd.py", line 66, in asyncio.run(main()) File "C:\Program Files\Python310\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete return future.result() File "C:\Temp\OnlyFans-master\start_ofd.py", line 52, in main api = await main_datascraper.start_datascraper(config, site_name) File "C:\Temp\OnlyFans-master\datascraper\main_datascraper.py", line 131, in start_datascraper await default(datascraper) File "C:\Temp\OnlyFans-master\datascraper\main_datascraper.py", line 101, in default await main_helper.process_jobs(datascraper, subscription_list, site_settings) File "C:\Temp\OnlyFans-master\helpers\main_helper.py", line 1026, in process_jobs await datascraper.start_datascraper(authed, subscription.username) File "C:\Temp\OnlyFans-master\modules\module_streamliner.py", line 84, in start_datascraper await self.prepare_scraper(subscription, content_type) File "C:\Temp\OnlyFans-master\modules\module_streamliner.py", line 232, in prepare_scraper master_set.extend(await self.datascraper.get_all_stories(subscription)) File "C:\Temp\OnlyFans-master\modules\onlyfans.py", line 388, in get_all_stories highlights = await subscription.get_highlights() File "C:\Temp\OnlyFans-master\apis\onlyfans\classes\user_model.py", line 283, in get_highlights results = [create_highlight(x) for x in results["data"]] KeyError: 'data' Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000001E3C8F96170> Traceback (most recent call last): File "C:\Program Files\Python310\lib\asyncio\proactor_events.py", line 116, in del self.close() File "C:\Program Files\Python310\lib\asyncio\proactor_events.py", line 108, in close self._loop.call_soon(self._call_connection_lost, None) File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 745, in call_soon self._check_closed() File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 510, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

bigscoops commented 2 years ago

File "D:\Data\OnlyFans-7.6.1\apis\onlyfans\classes\create_user.py", line 270, in get_highlights results = [create_highlight(x) for x in results.get("data")] TypeError: 'NoneType' object is not iterable

Can confirm, still getting this problem

I got this working by changing line 283 to results = [create_highlight(x) for x in results["list"]]

Fall0n commented 2 years ago

my line 270 is = identifier=self.id, global_limit=limit, global_offset=offset my line 283 is = result = handle_refresh(self, api_type)

do not have one thing results = [create_highlight(x) for x in results["list"]]

what version do i need change too make it work? or can some one post a working one..

Thanks

valdearg commented 2 years ago

my line 270 is = identifier=self.id, global_limit=limit, global_offset=offset my line 283 is = result = handle_refresh(self, api_type)

do not have one thing results = [create_highlight(x) for x in results["list"]]

what version do i need change too make it work? or can some one post a working one..

Thanks

https://github.com/DIGITALCRIMINALS/OnlyFans/issues/518#issuecomment-1227769081

Please read before posting.

Fall0n commented 2 years ago

my line 270 is = identifier=self.id, global_limit=limit, global_offset=offset my line 283 is = result = handle_refresh(self, api_type) do not have one thing results = [create_highlight(x) for x in results["list"]] what version do i need change too make it work? or can some one post a working one.. Thanks

#518 (comment)

Please read before posting.

dude did read that and that dont help.. OK?

valdearg commented 2 years ago

my line 270 is = identifier=self.id, global_limit=limit, global_offset=offset my line 283 is = result = handle_refresh(self, api_type) do not have one thing results = [create_highlight(x) for x in results["list"]] what version do i need change too make it work? or can some one post a working one.. Thanks

#518 (comment) Please read before posting.

dude did read that and that dont help.. OK?

You sure? The line hasn't changed: https://github.com/DIGITALCRIMINALS/OnlyFans/blob/master/apis/onlyfans/classes/user_model.py#L283

All you have to do is download the repo. NOT the release version, the whole repo. Then change that file on line 283

results = [create_highlight(x) for x in results["data"]] To results = [create_highlight(x) for x in results["list"]]

Fall0n commented 2 years ago

my line 270 is = identifier=self.id, global_limit=limit, global_offset=offset my line 283 is = result = handle_refresh(self, api_type) do not have one thing results = [create_highlight(x) for x in results["list"]] what version do i need change too make it work? or can some one post a working one.. Thanks

#518 (comment) Please read before posting.

dude did read that and that dont help.. OK?

You sure? The line hasn't changed: https://github.com/DIGITALCRIMINALS/OnlyFans/blob/master/apis/onlyfans/classes/user_model.py#L283

All you have to do is download the repo. NOT the release version, the whole repo. Then change that file on line 283

results = [create_highlight(x) for x in results["data"]] To results = [create_highlight(x) for x in results["list"]]

the master.zip i understand that like whole repo.

naodrej commented 2 years ago

@Fall0n upload the file you're looking at

Fall0n commented 2 years ago

upload the file you're looking at user_model.zip that my user_model.zip file..

that i get from

IMG_8893

what have i missunderstand

naodrej commented 2 years ago

upload the file you're looking at user_model.zip that my user_model.zip file..

that i get from

what have i missunderstand

My man, you're looking at the wrong file. You're editing the file from the Fansly folder when you should be doing it for the Onlyfans folder.

Fall0n commented 2 years ago

Fansly

upload the file you're looking at user_model.zip that my user_model.zip file..

that i get from what have i missunderstand

My man, you're looking at the wrong file. You're editing the file from the Fansly folder when you should be doing it for the Onlyfans folder.

seen all been talknig aboute Fansly dir all day. and now i find out im in the wrong did for the file i need edit lol. thanks

jsinfate commented 2 years ago

I was having the same issue with line 283, however, I found that if I ran updater.py this would change that line back to "data" even though I manually changed it to "list" as instructed.

I verified this by changing the line again to "list" then running start_ofd.py and I was able to successfully download. Hope this helps someone.