DIGITALCRIMINAL / ArchivedUltimaScraper

Scrape content from OnlyFans and Fansly
GNU General Public License v3.0
954 stars 39 forks source link

Limit issue with 7.6.1? -- TypeError: 'NoneType' object does not support item assignment #142

Closed lordoffools closed 2 years ago

lordoffools commented 2 years ago

Identical to #71. I'm having this issue after upgrading to 7.6.1 just now.

Exactly the same error as above. Prior to 7.6.1, I ran just fine with threads = -1.

I have over 1,000 profiles.

I just tried threads = 100, and still hit this issue.

Fails with threads = 10, as well.

Originally posted by @lordofcookies in https://github.com/DIGITALCRIMINALS/OnlyFans/issues/71#issuecomment-985973690

DIGITALCRIMINAL commented 2 years ago

Can't really do anything unless I have access to your account fix it since I can't replicate it

lordoffools commented 2 years ago

In theory, you just need the contents of my auth.json to be able to access and diagnose, right?

DIGITALCRIMINAL commented 2 years ago

Yeah it's exactly that

My email is on my original account

https://github.com/DIGITALCRIMINAL

lordoffools commented 2 years ago

Done.

DIGITALCRIMINAL commented 2 years ago

Were you using a VPN/Proxy?

CannotTouch commented 2 years ago

nope here

lordoffools commented 2 years ago

No VPN/Proxy.

Soulol36 commented 2 years ago

Hi, Have the same issue, Can help me? (TypeError: ´NoneType´ object is not iterable) Only Issue 2

m4rc1nt0sch commented 2 years ago

Happens to me as well, besides the difference it happens after the tenth Auth 404'ed error message:

screen
lordoffools commented 2 years ago

marcintosch

You seem to have two compounded issues. The 404 issue is a different one solved in #117.

The second issue appears to be similar to the one noted here.

@DIGITALCRIMINALS did you receive my auth.json from 12 hours ago via email, and was it helpful in allowing you to reproduce?

lordoffools commented 2 years ago

Hi, Have the same issue, Can help me? (TypeError: ´NoneType´ object is not iterable) Only Issue 2

From the error message you posted, that appears to be different from the one described in this bug.

TypeError: 'NoneType' object does not support item assignment

Soulol36 commented 2 years ago

Hi, Have the same issue, Can help me? (TypeError: ´NoneType´ object is not iterable) Only Issue 2

From the error message you posted, that appears to be different from the one described in this bug.

TypeError: 'NoneType' object does not support item assignment

Oh, true. Anyway I have had this error for a long time, but apparently, this time it seems to me this variant, but I have had this error since version EDIT: (7.6)

m4rc1nt0sch commented 2 years ago

marcintosch

You seem to have two compounded issues. The 404 issue is a different one solved in #117.

The second issue appears to be similar to the one noted here.

Thank you very much, buddy, haven't found the solution before, now everything works fine (and I don't get any NoneType TypeErrors anymore after installing the cert. Thanks a bunch 👍

CannotTouch commented 2 years ago

I have solved with old auth and config files following this one

#107 (comment)

this is a fix for the issue

and changing "active": active with "active": true in the auth file

lordoffools commented 2 years ago

Anyone having any luck diagnosing this? Anything I can help with in terms of data or attempts to reproduce using different settings?

lordoffools commented 2 years ago

If it helps, I only have this problem with OF, and not Fansly. Granted, if number of profiles is a factor, then that would explain it since Fansly is significantly less for me.

lordoffools commented 2 years ago

Anyone with >1,000 profiles in their account able to run this? Or are you also hitting this issue?

Trying to figure out if it's due to that or something else.

lordoffools commented 2 years ago

Still trying to resolve this issue. Adjusting settings, etc. Not having any luck.

Anyone have any insights into what this error means? As far as I can tell, it literally hits right as the script tries to pull the list of profiles.

lordoffools commented 2 years ago

I just realized that I never posted my own error here (it matches the above):

`Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10e0e67a0> Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10d7b01c0>

Traceback (most recent call last):   File "/start_ofd.py", line 104, in     asyncio.run(main())   File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run     return loop.run_until_complete(main)   File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete     return future.result()   File "/start_ofd.py", line 86, in main     api = await main_datascraper.start_datascraper(   File "/datascraper/main_datascraper.py", line 67, in start_datascraper     setup, subscriptions = await module.account_setup(   File "/modules/onlyfans.py", line 107, in account_setup     subscriptions += await manage_subscriptions(   File "/modules/onlyfans.py", line 1267, in manage_subscriptions     results = await authed.get_subscriptions(identifiers=identifiers, refresh=refresh)   File "/apis/onlyfans/classes/create_auth.py", line 312, in get_subscriptions     results += await asyncio.gather(tasks)   File "/apis/onlyfans/classes/create_auth.py", line 293, in multi     tasks = await asyncio.gather(tasks)   File "/apis/onlyfans/classes/create_auth.py", line 194, in get_user     response["session_manager"] = self.session_manager TypeError: 'NoneType' object does not support item assignment `

This starts right after a successful login (so I'm assuming it's when it tries to scrape the list of profiles). Then proceeds with a massively long list of:

Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10e0e67a0> Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10d7b01c0>

That repeats for a while until it hits the exception noted above in the traceback.

lordoffools commented 2 years ago

Anyone else having any luck with this?

I'm stuck. Not making any progress despite trying different things and researching the issue as best I can.

I feel like I'm yelling into the wind, at times, when I scroll up. :(

lordoffools commented 2 years ago

The rest of you who were having this issue above, are you still reproducing it?

Thanks!

lordoffools commented 2 years ago

FYI, still able to repro with latest pull from master.

lordoffools commented 2 years ago

Is there a way to set this to be verbose so it outputs where it's at with the profile scraping? Maybe it's hitting a bad profile?

Occasionally it fails after 15-30s of scraping, instead of immediately.

lordoffools commented 2 years ago

Anyone else still encountering this issue with a high number of subscribed profiles (paid or free, doesn't matter it seems).

I'm still reproducing this bug. Can't figure out how to fix.

@DIGITALCRIMINALS I can send you an updated auth.json if it'll help. Just let me know.

Thanks all.

lordoffools commented 2 years ago

Anyone else reproducing this issue?

lordoffools commented 2 years ago

Friendly nudge here.

Those of you who were encountering this issue earlier, are you still able to repro? I am. And I'm stuck. Can't figure out how to get past this.

DIGITALCRIMINAL commented 2 years ago

It's due to having too many requests, I was in the process of fixing this but I had to do something. I'd need a new authentication again to fix this.

The problem replies in the results2 variable which contains a list of users, but self.get_user will return None if there's a http error.

[OnlyFans | create_auth.py)]

for subscription in subscriptions:
    subscription["session_manager"] = self.session_manager
    if extra_info:
        task = asyncio.create_task(
            self.get_user(subscription["username"])
        )
        tasks.append(task)
results2 = await asyncio.gather(*tasks)
lordoffools commented 2 years ago

@DIGITALCRIMINALS glad to see you're Ok! Happy to send over a new auth.json to help out. Let me get re-authenticated and I'll email it over.

lordoffools commented 2 years ago

Sent. Please let me know if it didn't come through and/or if it doesn't work for you. @DIGITALCRIMINALS

DIGITALCRIMINAL commented 2 years ago

Got it, thanks

DIGITALCRIMINAL commented 2 years ago

I'm not encountering the error, is it still a problem for you?

lordoffools commented 2 years ago

Still reproduces.

Let me pull the latest code vs. 7.6.1.

lordoffools commented 2 years ago

Unfortunately, still reproduces with the latest pull.

DIGITALCRIMINAL commented 2 years ago

Try the latest commit

lordoffools commented 2 years ago

Fixed for me now with the latest commit.

Thanks, @DIGITALCRIMINALS!