vil / H4X-Tools

Open source toolkit for scraping, OSINT and more.
https://github.com/vil/H4X-Tools/wiki
GNU General Public License v3.0
320 stars 36 forks source link

H4X-Tools doesn't run after building it (Caused by twitter_scrape) #7

Closed vil closed 1 year ago

vil commented 1 year ago

After running setup.sh and typing h4xtools in terminal, it gives the following:

Traceback (most recent call last):
  File "h4xtools.py", line 25, in <module>
    from utils import email_search, search_username, ig_scrape, whois_lookup, webhook_spammer, ip_scanner, ip_lookup, \
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "utils/twitter_scraping.py", line 18, in <module>
    import snscrape.modules.twitter as snstwitter
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "snscrape/modules/__init__.py", line 17, in <module>
  File "snscrape/modules/__init__.py", line 13, in _import_modules
AttributeError: 'FrozenImporter' object has no attribute 'find_module'. Did you mean: 'exec_module'?
[380003] Failed to execute script 'h4xtools' due to unhandled exception!

H4X-Tools can be run normally tho using python3 h4xtools.py

vil commented 1 year ago

The issue is caused by twitter_scrape, also twitter_scrape doesn't work anymore. (Caused by the twitter api changes? and elons memeing)

snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=from%3Aelonmusk&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
Errors: blocked (403), blocked (403), blocked (403), blocked (403)

At this point I'm considering removing the whole twitter scrape tool.

@itsMyrto What do you think?

vil commented 1 year ago

Yep, after reading the issues for snscrape (https://github.com/JustAnotherArchivist/snscrape/issues/846), I guess its best to just remove the whole twitter scrape tool.

itsMyrto commented 1 year ago

Twitter announced that it would no longer allow any developer to use its APIs. I found out a few months after i contributed. If i knew that earlier i would probably scrape another social media. Anyways, I dont think you should keep something that doesn't work anymore and i really don't mind if you remove it. Another solution could be to remove it from the main branch and create a second branch to put it there just in case it works again in the future. But honestly, do what you feel its better for your project !

vil commented 1 year ago

Thanks for the reply, but yeah. Ima remove it for now and maybe replace it with something else? Since snscrape works with reddit and mastodon for example. If you want to help with it, feel free to contribute again!

Closing this issue now.