Threatingestor appears to be truncating text of tweets

[Hestat]

was trying to parse data from scumbots twitter feed and noticed that C2 data from empire was not being pulled. invesitgating using tweepy library noticed that the data was truncated as default setting still pull 140 chars while scumbots post is past that amount.

Example: https://twitter.com/ScumBots/status/1234682487856435208

2020-03-14 21:23:24.457 | DEBUG | threatingestor.sources:process_element:165 - Type breakdown: {'domain': 0, 'hash': 0, 'ipaddress': 0, 'task': 1, 'url': 0, 'yarasignature': 0} 2020-03-14 21:23:24.458 | DEBUG | threatingestor.sources:process_element:61 - Processing in source 'twitter-scumbots-empire

[shankaraman]

I am facing the same issue. In order to get the full tweet, we can use the flag: tweet_mode="extended" in the request and replace the text field as full_text from the JSON response.

ref EDIT : ref2

[shankaraman]

Other way to get the full tweet is the following:

Method 1: Inside your configuration file ( whatever_file_name.yml) add the tweet_mode='extended' flag:

• name: twitter-inquest-timeline module: twitter credentials: twitter-auth screen_name: blah-blah tweet_mode: extended

Method 2 (preferred): Directly update the parameter (kwargs) in the Plugin class (threatingestor/sources/twitter.py):

inside the constructor _ init _

  `  # Support for Full tweet.

    full_tweet_support = {'tweet_mode': 'extended'}
    kwargs.update(full_tweet_support) `

After, change the text field to full_text in threatingestor/sources/twitter.py :

tweets = [ { 'content': s['full_text'], 'id': s['id_str'], 'user': s['user']['screen_name'], 'entities': s.get('entities', {}), } for s in tweet_list ]

After making the modifications rebuild the project and run the tool again : you should get the full tweet. I tested and it works.

[cmmorrow]

Hello @Hestat, this issue has been addressed in release 1.0.0b7 which supports extended tweets. Please let me know if you're still seeming this problem after upgrading to 1.0.0b7.