minamotorin
commented
2 years ago Solution
The following patch works for me.
# This patch is WTFPL (http://www.wtfpl.net/txt/copying/) and no warranty.
diff --git a/twint/token.py b/twint/token.py
index ae66a24..2eedcee 100644
--- a/twint/token.py
+++ b/twint/token.py
@@ -65,5 +65,30 @@ class Token:
logme.debug('Found guest token in HTML')
self.config.Guest_token = str(match.group(1))
else:
- self.config.Guest_token = None
- raise RefreshTokenException('Could not find the Guest token in HTML')
+ headers = {
+ 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
+ 'authority': 'api.twitter.com',
+ 'content-length': '0',
+ 'authorization': self.config.Bearer_token,
+ 'x-twitter-client-language': 'en',
+ 'x-csrf-token': res.cookies.get("ct0"),
+ 'x-twitter-active-user': 'yes',
+ 'content-type': 'application/x-www-form-urlencoded',
+ 'accept': '*/*',
+ 'sec-gpc': '1',
+ 'origin': 'https://twitter.com',
+ 'sec-fetch-site': 'same-site',
+ 'sec-fetch-mode': 'cors',
+ 'sec-fetch-dest': 'empty',
+ 'referer': 'https://twitter.com/',
+ 'accept-language': 'en-US',
+ }
+ self._session.headers.update(headers)
+ req = self._session.prepare_request(requests.Request('POST', 'https://api.twitter.com/1.1/guest/activate.json'))
+ res = self._session.send(req, allow_redirects=True, timeout=self._timeout)
+ match = re.search(r'{"guest_token":"(\d+)"}', res.text)
+ if match:
+ logme.debug('Found guest token in JSON')
+ self.config.Guest_token = str(match.group(1))
+ else:
+ self.config.Guest_token = None
+ raise RefreshTokenException('Could not find the Guest token in JSON')I don't understand session of requests, so the code may be not good. I hope someone rewrite the patch better and create a pull request.
About the problem
In my environment, this problem has recently begun to occur. This doesn't happen every time, so if you are lucky, you don't get the error.
The cause is literally literally that twint could not find the Guest token in HTML. Actually, sometimes token isn't included in HTML recently.
#!/usr/bin/env python3
# This program is WTFPL.
import requests
res = requests.get('https://twitter.com')
print(res.text.split('\n')[-1])twint require the result of running the above code is })();</script><script nonce="VALUE">document.cookie = decodeURIComponent("gt=VALUE; Max-Age=VALUE; Domain=.twitter.com; Path=/; Secure");</script>.
However, sometimes the result is only })();</script> and missing the Guest token.
About the solution
In my patch, twint get the Guest token from https://api.twitter.com/1.1/guest/activate.json if could not find the one. I referred to the code of gallery-dl.
These are the steps I took to install twint and get try and get it to work on 2 versions of Linux and 1 Deb. I have spent 9 hours trying my best, but no fix anywhere, none at all, not one.
Unfortunately, twint hasn't been updated for a long time. If the problem is new, updating will not solve the problem.
In this case, the problem is too new to find a solution. However, the problem may not happen in other environments, so I want reports.
yvessaintlaureint
LinqLover
aaabbbbbbb
prhbrt
rangersmyth74
Silvthril
mlkorra
jonathanmc
Hoefnix
ElizabethCappon
irisdemented
codeghees
Florettt
OrionLi545
universityofkalilinux
I really love this app and I have only used it once. But now I want to use it and I can't. I have searched so much that my eyes are sore, why am I getting these errors and why can't I find a fix for it!! hehe...
Linux Kali-ROG 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux
Please help not just me but anyone who searches the 400 pages of information to find an answer. 9 hours of work to try this, the time I will never get back and I just want to chuck in Linux altogether and do anything else.
pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;These are the steps I took to install twint and get try and get it to work on 2 versions of Linux and 1 Deb. I have spent 9 hours trying my best, but no fix anywhere, none at all, not one.
What errors am I getting, all listed below
1: Install on Kali. This did not work for me.
Linux Kali-ROG 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux
Instructions for install : https://github.com/twintproject/twint
Try and run these commands to see if it works.
2: Install on Kali AWS. This did not work for me.
Linux kali 5.15.0-kali2-cloud-amd64 #1 SMP Debian 5.15.5-2kali2 (2021-12-22) x86_64 GNU/Linux
It seems that twint and AWS do not work due to
Instructions for install : https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
Try and run these commands to see if it works.
3: Install on Google Cloud Shell and This worked for me.
Lightbox Linux cs-899333161534-default-boost-h8jtm 5.10.68+ #1 SMP Wed Dec 1 10:07:21 UTC 2021 x86_64 GNU/Linux
Instructions for install: https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
FYI 1 - Google didn't like the command git clone --depth=1
So I removed it and it installed.
FY 2 - Using pip3 install twint command said all was installed, but I added sudo
$ sudo git clone https://github.com/twintproject/twint.git $ ls $ cd twint $ ls $ sudo pip3 install . -r requirements.txt $ sudo pip3 install twint
Ran commands to see if it works.
I was able to run twint until I logged out and logged back in, it didn't work anymore.
The Kali twint install journey.
My Install on Kali Linux.
┌──(kali㉿kali)-[~] └─$ git clone --depth=1 https://github.com/twintproject/twint.git Cloning into 'twint'... remote: Enumerating objects: 47, done. remote: Counting objects: 100% (47/47), done. remote: Compressing objects: 100% (44/44), done. remote: Total 47 (delta 3), reused 14 (delta 0), pack-reused 0 Receiving objects: 100% (47/47), 42.95 KiB | 1.59 MiB/s, done. Resolving deltas: 100% (3/3), done.
┌──(kali㉿kali)-[~] └─$ ls Desktop Documents Downloads Music Pictures Public Templates Videos bash.txt nmap twint
┌──(kali㉿kali)-[~] └─$ cd twint
┌──(kali㉿kali)-[~/twint] └─$cd twint
┌──(kali㉿kali)-[~/twint] └─$ pip3 install . -r requirements.txt Command 'pip3' not found, but can be installed with: sudo apt install python3-pip Do you want to install it? (N/y)y sudo apt install python3-pip
┌──(kali㉿kali)-[~/twint] └─$ pip3 install . -r requirements.txt
WARNING: The script twint is installed in '/home/kali/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed aiohttp-socks-0.4.1 cchardet-2.1.7 dataclasses-0.6 elasticsearch-7.16.2 fake-useragent-0.1.11 geographiclib-1.52 geopy-2.2.0 googletransx-2.4.2 pandas-1.3.5 schedule-1.1.0 twint-2.1.21
(I added two commands for the path later on in the install, see far below).
┌──(kali㉿kali)-[~/.local] └─$ cd bin
┌──(kali㉿kali)-[~/.local/bin] └─$ ls cchardetect translate twint
┌──(kali㉿kali)-[~/.local/bin] └─$ twint -u networkchuck Traceback (most recent call last): File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
┌──(kali㉿kali)-[~/.local/bin] └─$ echo $SHELL
/bin/zsh
┌──(kali㉿kali)-[~/.local/bin] └─$ bash
┌──(kali㉿kali)-[~/.local/bin] └─$ myip Command 'myip' not found, did you mean: command 'mzip' from deb mtools Try: sudo apt install
┌──(kali㉿kali)-[~/.local/bin] └─$ twint -u networkchuck -s "raspberry pi" Traceback (most recent call last): File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
──(kali㉿kali)-[~/.local/bin] └─$ cd ..
┌──(kali㉿kali)-[~/.local] └─$ cd.. cd..: command not found
┌──(kali㉿kali)-[~/.local] └─$ cd ..
┌──(kali㉿kali)-[~] └─$
┌──(kali㉿kali)-[~] └─$ twint -u networkchuck -s "raspberry pi" Command 'twint' not found, did you mean: command 'twine' from deb twine Try: sudo apt install
┌──(kali㉿kali)-[~/twint] └─$ python3 Python 3.9.9 (main, Dec 16 2021, 23:13:29) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
┌──(kali㉿kali)-[~/twint] └─$ twint -u networkchuck -s "raspberry pi" Command 'twint' not found, did you mean: command 'twine' from deb twine Try: sudo apt install
┌──(kali㉿kali)-[~] └─$ sudo apt update && sudo apt upgrade
┌──(kali㉿kali)-[~] └─$ systemctl reboot -i
┌──(kali㉿kali)-[~/twint] └─$cd twint
┌──(kali㉿kali)-[~/twint] └─$ chmod +x setup.py
┌──(kali㉿kali)-[~/twint] └─$ python3 setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help
error: no commands supplied ──(kali㉿kali)-[~/twint] └─$ python3 setup.py --help
Common commands: (see '--help-commands' for more)
setup.py build will build the package underneath 'build/' setup.py install will install the package
┌──(kali㉿kali)-[~/twint] └─$ sudo python3 setup.py install
running install
┌──(kali㉿kali)-[~/twint] └─$ sudo python3 setup.py build
running build running build_py
┌──(kali㉿kali)-[~/twint] └─$ twint -u networkchuck -s "raspberry pi" Traceback (most recent call last): File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
Research Issue.
Fix 1: I searched on Google with the error and found,
https://github.com/twintproject/twint/issues/1071
Asked to run the command below to check if there was a .local/bin folder created.
$ echo $PATH | tr -s ":" "\n" | sort
┌──(kali㉿kali)-[~/Downloads/twint] └─$ echo $PATH | tr -s ":" "\n" | sort 1 ⨯ /bin /home/kali/.dotnet/tools /home/kali/.local/bin /sbin /usr/bin /usr/games /usr/local/bin /usr/local/games /usr/local/sbin /usr/sbin
The advice was to add these two lines to the .zshrc script as I am using zsh shell. I am having the same issue in bash, so I don't think it matters at all for the moment.
$ export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin"
$ export PATH="$PATH:$PYTHON_BIN_PATH"
This method didn't work for me. Still have the error.
Fix2: $ pip3 install yarl --force-reinstall --no-cache-dir
https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1
A fix was mentioned by a user - "I had the same issue with python3.7 and resolve it by reinstalling yarl, like this way:"
$ pip3 install yarl --force-reinstall --no-cache-dir
This method didn't work for me.
Fix3: To uninstall and reinstall twint with the upgrade.
https://github.com/twintproject/twint/issues/915
This method didn't work for me.
Fix4: Install on Google Cloud Shell.
Google Cloud New build
This worked for me but once you leave the session, twint seems to be causing the same issue as I had on the Kali machine.
twint -u networkchuck –limit 20 (Did not recognise -limit 20)
twint -u networkchuck -s “raspberry pi”
New Error message.
$ twint -u networkchuck -s crypto -o rightnow.json–json
Traceback (most recent call last): File "/home/rangersmyth_74/.local/bin/twint", line 10, in
sys.exit(run_as_command())
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 339, in run_as_command
main()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 330, in main
run.Search(c)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 410, in Search
run(config, callback)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 329, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 36, in init
self.token.refresh()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/token.py", line 69, in refresh
raise RefreshTokenException('Could not find the Guest token in HTML')
twint.token.RefreshTokenException: Could not find the Guest token in HTML
Searching for twint.token
https://github.com/twintproject/twint/pull/1146 Fix 5. Add to PATH. https://issueexplorer.com/issue/twintproject/twint/1189
"" MikeTheScriptKid wrote this answer on 2021-05-08 "" So I finally got the twint command working in Kali. I added the following to /etc/environment /your_user/.local/bin
"" HarryWestFord wrote this answer on 2021-05-10 "" "sorry for being such a newbie but how excalty do you add this on the command line?
"" hackingbutlegal wrote this answer on 2021-05-15 ""
nano /etc/environment
This didn't work for me.
Found a fix to sort out the token problem
SCRIPT from >> https://github.com/twintproject/twint/pull/1146
import re import time import logging as logme import requests
class TokenExpiryException(Exception): def init(self, msg): super().init(msg)
class RefreshTokenException(Exception): def init(self, msg): super().init(msg)
class Token: def init(self, config): self._session = requests.Session() self._session.headers.update( {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}) self.config = config self._proxies = self._set_proxies() self._retries = 5 self._timeout = 10 self.url = 'https://twitter.com'
I ran this script and got back nothing, no error, but when I did a search it didn't work.
Back to the Drawing Board.
References:
https://github.com/twintproject/twint/wiki/Setup
https://github.com/twintproject/twint/wiki/Configuration
https://github.com/twintproject/twint/issues/468
https://github.com/twintproject/twint/issues/915
https://github.com/twintproject/twint/pull/917
https://github.com/twintproject/twint/pull/944
https://github.com/twintproject/twint/issues/980
https://github.com/twintproject/twint/issues/1114 - Has a fix to upgrade twint && I also repored my issue here.
https://github.com/twintproject/twint/pull/1146 - Token Fix
https://github.com/Altimis/Scweet
http://saka.docsio.net/66892325/scrape-join-dates-user-info-from-a-list-csv-of-twitter-users
http://saka.docsio.net/64747304/twint-python-library-is-causing-exception-for-search-qeruy
https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
https://github.com/himanshudabas/twint/tree/twint-fixes
https://www.kaggle.com/zelinngilo/punya-nadia?scriptVersionId=57423357
https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1
https://www.bountysource.com/teams/tweep/issues?tracker_ids=65007358
https://issueexplorer.com/issue/twintproject/twint/1266