I can't get any values by using Scraper

[berkingurcan]

I use cookies and Account is working well, I can reply or get some content. However, when I try to use Scraper and its methods, I cannot get any value all return values are empty.

I have used:

scraper.trends()
# or
tweets_and_replies = scraper.tweets_and_replies([1096830491276259330], limit=limit)
# or
scraper.rate_limits

I am getting:

Cannot parse JSON response 'NoneType' object has no attribute 'json'

or empty dict for scraper.rate_limits

Are we sure scraper is working? Or am I missing something?

[trevorhobenshield]

This almost always boils down to cookies being expired, pending security challenges, or a blocked account. I've updated this error message to provide more details on what to do in this circumstance. The solution is to log-in using the browser and confirm everything is OK, and copy cookies from that session and try again.

[berkingurcan]

No, it is not blocked or something like that. I use fresh cookies, and it has no problem, everything is okay also in browser. Also as I said before, Account methods is working.

[trevorhobenshield]

I can confirm it's currently working:

[CyrilDesch]

Not working for me too

[ittekikun]

I have the same problem. I logged in anew and used a new cookie, but no improvement.

raw_tweets = Scraper(cookies=COOKIE_PATH).tweets([TARGET_USER], limit=200) Please let me know if I am missing any information necessary to resolve this issue.

[berkingurcan]

I have tried with another accounts cookies(got from another computer also) but still same.

[TMDR]

I don't know if we're facing the same issue, but as of lately, I noticed that Twitter has been able to detect automated requests and lock it out via Arkose challenge. I even tried using my own, verified, personal account, It still was able to. Cookies don't last much anymore before an Arkose challenge is needed, but I still don't know how long does it last after doing the challenge. I know this is a new issue because I ran automated tasks for months via this library and there was not the slightest hiccup, only as of lately I needed to generate new cookies almost every day. I wonder if randomizing the behavior would lead to better results?

[ittekikun]

This is very puzzling, but when I added "debug=1" to the argument, it now works correctly.

scraper = Scraper(cookies=COOKIE_PATH, debug=1)

[rrasch]

Hi @ittekikun I also solved the issue by setting debug=1 in the constructor.

It wasn't a cookie issue for me.

I tracked it down to the Scraper._query() method. It looks like the logger isn't initialized when debug is set to 0 causing an AttributeError exception when calling self.logger.debug().

Here's the exact exception:

AttributeError("'NoneType' object has no attribute 'debug'")

As a workaround, I made the following change:

--- scraper.py.old  2024-03-30 15:37:17.000000000 -0400
+++ scraper.py  2024-04-10 22:53:45.700505971 -0400
@@ -601,7 +601,8 @@
             fn_name = sys._getframe(9).f_code.co_name
             self.rate_limits[fn_name] = {'_endpoint': name} | {k: int(v) for k, v in r.headers.items() if 'rate-limit' in k}
         except Exception as e:
-            self.logger.debug(f'{e}')
+            if self.debug:
+                self.logger.debug(f'{e}')

         if self.debug:
             log(self.logger, self.debug, r)