search

rdbis / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
0 stars 0 forks source link

[BUG] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #4

Open rdbis opened 1 year ago

rdbis commented 1 year ago

twint crashes when twint.run.Profile testcase is executed.

Command Ran

python test.py

Description of Issue

running test.py results in twint crash: python.exe J:/twint-clean/test.py [+] Beginning vanilla test in <function Profile at 0x000001E6D6E527A0> Traceback (most recent call last): File "J:\twint-clean\test.py", line 92, in main() File "J:\twint-clean\test.py", line 78, in main test(c, run) File "J:\twint-clean\test.py", line 11, in test_reg run(c) File "J:\twint-clean\twint\run.py", line 398, in Profile run(config) File "J:\twint-clean\twint\run.py", line 329, in run get_event_loop().run_until_complete(Twint(config).main(callback)) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 649, in run_until_complete return future.result() File "J:\twint-clean\twint\run.py", line 235, in main await task File "J:\twint-clean\twint\run.py", line 283, in run await self.profile() File "J:\twint-clean\twint\run.py", line 210, in profile await self.Feed() File "J:\twint-clean\twint\run.py", line 62, in Feed response = await get.RequestUrl(self.config, self.init) File "J:\twint-clean\twint\get.py", line 135, in RequestUrl response = await Request(_url, params=params, connector=_connector, headers=_headers) File "J:\twint-clean\twint\get.py", line 161, in Request return await Response(session, _url, params) File "J:\twint-clean\twint\get.py", line 170, in Response raise TokenExpiryException(loads(resp)['errors'][0]['message']) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64qbz5n2kfra8p0\lib\json\init.py", line 346, in loads return _default_decoder.decode(s) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Process finished with exit code 1

rdbis commented 1 year ago

after debug it looks like there at least two problems related to this issue:

  1. observed crash is caused by incorrect handling of "Too many requests" response from twitter, see code below. Crash is triggered by the access to empty resp var passed as parameter to TokenExpiryException.

    async def Response(session, _url, params=None): logme.debug(name + ':Response') with timeout(120): async with session.get(_url, ssl=True, params=params, proxy=httpproxy) as response: resp = await response.text() if response.status == 429: # 429 implies Too many requests i.e. Rate Limit Exceeded raise TokenExpiryException(loads(resp)['errors'][0]['message']) return resp

  2. The second more important issue is to fix the problem that triggers the 429 (Too many requests) response from twitter. Looks like, twitter responds with 429 code when asked with SearchProfile function, using obsolete API (timeline). New issue will be opened for this case.