bobsyourmom commented 4 years ago

Running tram.py fails once it gets to loading taxii feeds

Any relevant information regarding the bug, such as error messages and the steps taken to produce the error. (base) FLC-SA-WDG8WL:tram xxx$ sudo python tram.py Password: [nltk_data] Downloading package punkt to /Users/xxx/nltk_data... [nltk_data] Package punkt is already up-to-date! INFO:root:Welcome to TRAM DEBUG:asyncio:Using selector: KqueueSelector INFO:root:Downloading ATT&CK data from STIX/TAXII... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cti-taxii.mitre.org:443 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/ HTTP/1.1" 200 249 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=attack-pattern HTTP/1.1" 200 1614940 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=intrusion-set HTTP/1.1" 200 223280 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=malware HTTP/1.1" 200 492177 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=tool HTTP/1.1" 200 72769 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=relationship HTTP/1.1" 200 8028304 CRITICAL:root:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! COULD NOT CONNECT TO TAXII SERVERS: 'description' PLEASE UTILIZE THE OFFLINE CAPABILITY FLAG "-FF" FOR OFFLINE DATABASE BUILDING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ERROR:asyncio:Task exception was never retrieved future: <Task finished coro=<background_tasks() done, defined at tram.py:23> exception=SystemExit()> Traceback (most recent call last): File "tram.py", line 36, in background_tasks await data_svc.insert_attack_stix_data() File "/Users/xxx/TRAM/tram/service/data_svc.py", line 54, in insert_attack_stix_data "description": i['description'].replace('', '').replace('', '').replace( File "/Users/xxx/opt/anaconda3/lib/python3.7/site-packages/stix2/base.py", line 195, in getitem return self._inner[key] KeyError: 'description'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/xxx/opt/anaconda3/lib/python3.7/asyncio/base_events.py", line 566, in run_until_complete self.run_forever() File "/Users/xxx/opt/anaconda3/lib/python3.7/asyncio/base_events.py", line 534, in run_forever self._run_once() File "/Users/xxx/opt/anaconda3/lib/python3.7/asyncio/base_events.py", line 1771, in _run_once handle._run() File "/Users/xxx/opt/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "tram.py", line 42, in background_tasks sys.exit() SystemExit

KadeMorton commented 4 years ago

I'm also receiving the same error as the original post after following the instructions in the README and running python3 tram.py

I noticed that if I run sudo python3 rather than the above error I receive the below: Traceback (most recent call last): File "tram.py", line 7, in import aiohttp_jinja2 ModuleNotFoundError: No module named 'aiohttp_jinja2'

This is using an Ubuntu 20.04 VM on VirtualBox

markeaimark commented 4 years ago

This may be a networking issue. Are you able to access the taxii server with this script?:

from stix2 import TAXIICollectionSource, Filter
from taxii2client.v20 import Server, Collection
collections = {
    "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e",
    "pre_attack": "062767bd-02d2-4b72-84ba-56caef0f8658",
    "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b"
}
server = Server("https://cti-taxii.mitre.org/taxii/")
api_root = server.api_roots[0]
for collection in api_root.collections:
    print(collection.title + ": " + collection.id)
collection = Collection(f"https://cti-taxii.mitre.org/stix/collections/{collections['enterprise_attack']}/")
tc_src = TAXIICollectionSource(collection)
result = tc_src.query([
    Filter("type", "=", "marking-definition")
])
print(result)
def get_technique_by_name(thesrc, name):
    filt = [
        Filter('type', '=', 'attack-pattern'),
        Filter('name', '=', name)
    ]
    return thesrc.query(filt)
print(get_technique_by_name(tc_src, 'SSH Authorized Keys'))

KadeMorton commented 4 years ago

Thanks. I ran the above script, it was throwing errors. I changed the module name from taxii2client.v20 to taxii2client, that allowed it to run. It came back with information on the attack pattern for SSH Authorized Keys. As I can access the server it doesn't appear to be a networking issue.

jecarr commented 4 years ago

Hello!

I came across both errors so thought I'd share how I got on.

Original error: Debugging the attacks array in the method insert_attack_stix_data() showed out of many attack objects looped in data_svc.py line 51, one did not have a description (hence the KeyError for 'description'). Pull request #61 created.

Some extra info on errors I had in the run up to fixing this in case it helps anyone:

My setup: Using python3.7* in a virtual environment via PyCharm (installed on Ubuntu)

*this is because python is Python 2.7 and python3 is Python 3.6 for me, replace python3.7 with whatever command/interpreter you are using

Ran python3.7 tram.py (not with sudo)

Got ModuleNotFoundError: No module named 'aiohttp_jinja2' output described earlier
Had to setup slightly differently, instead of pip install -r requirements.txt did python3.7 -m pip install -r requirements.txt
(For more info on why this is important for an environment with different Python interpreters, see here)

Errors

I initially was doing python3.7 -m pip install <module name> for every module which python3.7 tram.py error'ed on and had some errors. Best to install using the list in the requirements file but if this is the approach you are taking, some issues may arise where

pip is installing a Python2-targeted module
- For example python3.7 -m pip install newspaper outputted for me WARNING! You are attempting to install newspaper's python2 repository on python3. PLEASE RUN '$ pip3 install newspaper3k' for python3 or '$ pip install newspaper' for python2
- Fixed by running python3.7 -m pip install newspaper3k (i.e. be wary of module names, again this would be better to depend on the requirements file for this)
...File "/usr/lib/python3/dist-packages/PIL/Image.py", line 60, in <module> from . import _imaging as core ImportError: cannot import name '_imaging' from 'PIL' (/usr/lib/python3/dist-packages/PIL/__init__.py)
- Suggestion from here https://github.com/OlafenwaMoses/ImageAI/issues/171#issuecomment-658309368 worked so I did
  - sudo python3.7 -m pip uninstall Pillow (sudo as this is uninstalling)
  - python3.7 -m pip install Pillow
python3.7 tram.py then gave LookupError: ********************************************************************** Resource punkt not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('punkt')
- As suggested in the error, I ran python3.7 (change dependent on your interpreter) and ran the two lines separately
- Edit: This is fixed in pull request #49

Finally, I then received the initial error from this thread and debugged that. After my fix, it took nearly 30mins for it to load but this could vary from machine to machine (so hopefully you'll see the main page in time!)

Hope it helps!

jecarr commented 4 years ago

Hey @markeaimark - any chance we can progress on this ticket by either getting feedback on the pull request and/or getting it merged in?

Edit - @isaisabel - any thoughts on this please?

robomotic commented 4 years ago

Yes please I have the same issue.

isaisabel commented 3 years ago

Hi all,

Sorry for the lack of communication here. The TRAM developers have been focusing on other projects but will be returning to TRAM soon. We haven't forgotten about it, we've just been busy with other stuff.

ghost commented 3 years ago

Hi all,

Sorry for the lack of communication here. The TRAM developers have been focusing on other projects but will be returning to TRAM soon. We haven't forgotten about it, we've just been busy with other stuff.

Is there any news about this issue? I can't get tram to run.

jecarr commented 3 years ago

Hey @timoliciouz - is it this issue in particular you are having trouble with (KeyError: 'description')?

If so, my code change in #61 fixed the error for me.

Else if it's a different error: first, check the other Issues here to see if any cover your scenario. If no open Issues do so, open a new ticket with the console output/details of how tram is not running for you.

ghost commented 3 years ago

Hey @jecarr thank you so much, I tried that code snipped and after fixing some dependencies TRAM is starting now. I do not see any content when I want to analyze one of the reports, but at least I can add some reports to the queue. Unfortunately the tool is not working I guess: I am waiting for a report of the queue being ready for athe review but nothing happens since about 20 minutes now. This is really frustrating.

khlaifiabilel commented 3 years ago

Greetings , I do have probably the same problem when i run the Python tram.py command

/home/billal/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject return f(*args, **kwds) INFO:root:Welcome to TRAM DEBUG:asyncio:Using selector: EpollSelector INFO:root:Downloading ATT&CK data from STIX/TAXII... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cti-taxii.mitre.org:443 DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/ HTTP/1.1" 406 122 ERROR:root:Server Response: 406 Client Error If you are trying to contact a TAXII 2.0 Server use 'from taxii2client.v20 import X'. If you are trying to contact a TAXII 2.1 Server use 'from taxii2client.v21 import X' CRITICAL:root:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! COULD NOT CONNECT TO TAXII SERVERS: The underlying TAXII Collection resource defined in the supplied TAXII Collection object provided could not be reached. Recieved error: "406 Client Error: Not Acceptable for url: https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/" PLEASE UTILIZE THE OFFLINE CAPABILITY FLAG "-FF" FOR OFFLINE DATABASE BUILDING !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ERROR:asyncio:Task exception was never retrieved future: <Task finished coro=<background_tasks() done, defined at tram.py:21> exception=SystemExit()> Traceback (most recent call last): File "/home/billal/anaconda3/lib/python3.7/site-packages/stix2/datastore/taxii.py", line 155, in init if collection.can_read: File "/home/billal/anaconda3/lib/python3.7/site-packages/taxii2client/v21/init.py", line 248, in can_read self._ensure_loaded() File "/home/billal/anaconda3/lib/python3.7/site-packages/taxii2client/v21/init.py", line 317, in _ensure_loaded self.refresh() File "/home/billal/anaconda3/lib/python3.7/site-packages/taxii2client/v21/init.py", line 331, in refresh response = self.__raw = self._conn.get(self.url, headers={"Accept": accept}) File "/home/billal/anaconda3/lib/python3.7/site-packages/taxii2client/common.py", line 309, in get raise e File "/home/billal/anaconda3/lib/python3.7/site-packages/taxii2client/common.py", line 299, in get resp.raise_for_status() File "/home/billal/anaconda3/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 406 Client Error: Not Acceptable for url: https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tram.py", line 34, in background_tasks await data_svc.insert_attack_stix_data() File "/home/billal/Documents/Github/tram/service/data_svc.py", line 42, in insert_attack_stix_data tc_source = TAXIICollectionSource(collection) File "/home/billal/anaconda3/lib/python3.7/site-packages/stix2/datastore/taxii.py", line 166, in init " Collection object provided could not be reached. Recieved error:", e, stix2.datastore.DataSourceError: The underlying TAXII Collection resource defined in the supplied TAXII Collection object provided could not be reached. Recieved error: "406 Client Error: Not Acceptable for url: https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/"

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/billal/anaconda3/lib/python3.7/asyncio/base_events.py", line 570, in run_until_complete self.run_forever() File "/home/billal/anaconda3/lib/python3.7/asyncio/base_events.py", line 538, in run_forever self._run_once() File "/home/billal/anaconda3/lib/python3.7/asyncio/base_events.py", line 1782, in _run_once handle._run() File "/home/billal/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "tram.py", line 40, in background_tasks sys.exit() SystemExit

jecarr commented 3 years ago

Hey @khlaifiabilel - similar error stack traces, yes, but the core of your issue is actually covered in #69.

This fix should help: https://github.com/mitre-attack/tram/pull/61/commits/8f68e77#diff-0746836447d6aec88640ed0beff7b7e9d85354c9f503f4c622b1d239c1b3a21e

deadbits commented 3 years ago

I am still hitting this same issue on the Description key and I'm a bit unclear on how to actually resolve this from this thread. Any help here would be great. Tram won't run at all past this point.

» python tram.py                                                                                                                                                           [11:47:25]
tram.py:22: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
  async def background_tasks(taxii_local='online', build=False, json_file=None):
tram.py:46: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
  async def init(host, port):
INFO:root:Welcome to TRAM
DEBUG:asyncio:Using selector: KqueueSelector
INFO:root:Downloading ATT&CK data from STIX/TAXII...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cti-taxii.mitre.org:443
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/ HTTP/1.1" 200 249
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=attack-pattern HTTP/1.1" 200 2086477
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=intrusion-set HTTP/1.1" 200 283775
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=malware HTTP/1.1" 200 617134
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=tool HTTP/1.1" 200 85278
DEBUG:urllib3.connectionpool:https://cti-taxii.mitre.org:443 "GET /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=relationship HTTP/1.1" 200 10831710
CRITICAL:root:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
COULD NOT CONNECT TO TAXII SERVERS: 'description'
PLEASE UTILIZE THE OFFLINE CAPABILITY FLAG "-FF" FOR OFFLINE DATABASE BUILDING
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-1' coro=<background_tasks() done, defined at tram.py:21> exception=SystemExit()>
Traceback (most recent call last):
  File "tram.py", line 34, in background_tasks
    await data_svc.insert_attack_stix_data()
  File "/Users/aswanda/Research/projects/intel-entity-extraction/tram/service/data_svc.py", line 54, in insert_attack_stix_data
    "description": i['description'].replace('<code>', '').replace('</code>', '').replace(
  File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/stix2/base.py", line 195, in __getitem__
    return self._inner[key]
KeyError: 'description'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "tram.py", line 40, in background_tasks
    sys.exit()
SystemExit

jecarr commented 3 years ago

Hey @deadbits - as you have a KeyError for 'description', there are two places in your copy of the codebase - in the file service/data_svc.py - where I'd suggest tweaking:

archanchoudhury commented 3 years ago

Hey @jecarr , I am stuck with the exact error like @deadbits . I see in both the branches you mentioned the changes on line 55 and 64 looks to be same, and the 2nd branch has other changes as well which are not there in the first branch. Also, config and tram.py files have also been tweaked on the 2nd branch(7d357fd), So could you please suggest the tweaking is required only on the data_svc.py file or the other files as well like in 7d357fd? I have taken the codes from https://github.com/mitre-attack/tram/tree/7d357fd5a6c0435ada9c60e58d17ce887b7b4689 branch and got the same issue with 'description'. I believe the changes in 060b4f2 has been covered in 7d357fd as well with some addition. Please suggest!

Thanks in advanced!

jecarr commented 3 years ago

Hey @archanchoudhury - sorry to hear the error is still persisting.

No need to worry about the other files changed in 7d357fd: the config changes are file comments (so would be ignored); and the tram.py changes are just replacing parameters with variables (so the functionality would not be affected).

I listed 060b4f2 because it has the full line

"description": i['description'].replace('<code>', '').replace('</code>', '').replace('\n', '').encode('ascii', 'ignore').decode('ascii') if hasattr(i, "description") else 'No description provided',
"similar_words": ...

whereas 7d357fd just tweaked some spacing on that line. You are right, if you use https://github.com/mitre-attack/tram/tree/7d357fd5a6c0435ada9c60e58d17ce887b7b4689 this does have commit 060b4f2.

I switched to that commit but wasn't able to replicate the issue. As you are still experiencing this issue, let's double check some things:

The error block posted by deadbits contains this part:
```
COULD NOT CONNECT TO TAXII SERVERS: 'description'
...
Traceback (most recent call last):
File "tram.py", line 34, in background_tasks
await data_svc.insert_attack_stix_data()
File "/Users/aswanda/Research/projects/intel-entity-extraction/tram/service/data_svc.py", line 54, in insert_attack_stix_data
"description": i['description'].replace('<code>', '').replace('</code>', '').replace(
File "/Users/aswanda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/stix2/base.py", line 195, in __getitem__
return self._inner[key]
KeyError: 'description'
```
If this is the exact error you are having, please note the following:
- File "...data_svc.py", line 54, in insert_attack_stix_data
- Go to where insert_attack_stix_data is in data_svc.py
- Look for the description-replace line printed in the error (in this case, it is on line 54 and begins with "description": i['description'].replace('<code>', '').replace('</code>', '').replace()
- Check the description-replace line has the fix at the end (i.e. is if hasattr(i, "description") else 'No description provided' added at the end of the description-replace line?)
If the above doesn't help and given this is a 'description' error:
- Observe your Traceback error
- Look for where data_svc.py is mentioned
- Next to this, make note of the line number and method (is it insert_attack_json_data for example?)
- Search the method for the line which is using 'description' and follow the previous steps (ensuring there is that if-else line at the end of the culprit line)
Edit - I went through 'description' references in data_svc.py to try and highlight where else this error may be occurring: arachne-threat-intel/thread@31b060d. The line numbers may be different to yours as I've heavily edited that forked repo so when looking at the commit, just refer to the method and which surrounding lines look similar.

If you are still having issues, feel free to paste your stack trace (error block like what deadbits posted). The output of git diff data_svc.py will also help
- If the output from this is long, you may have to scroll until the bottom of your terminal doesn't have a :. You will then have to hit 'q' to exit the command.
- The git diff command may error if you are not in the right directory (so cd into tram/service)

archanchoudhury commented 3 years ago

Hey @jecarr , Thank you so much for taking time and detailed explanation. Finally TRAM is up and RUNNING!! Awesome work!!

ioannoug commented 3 years ago

Hi @jecarr I am having trouble following your advice, particularly the

Check the description-replace line has the fix at the end (i.e. is if hasattr(i, "description") else 'No description provided' added at the end of the description-replace line?)

That code block is now: Where should the if hasattr(i, "description") else 'No description provided' part go to?

Cheers George

I have found a silly workaround but I am still curious!

jecarr commented 3 years ago

Hey @ioannoug, glad to hear you got a workaround. To answer your question, I ended up simplifying that code change further.

I no longer did if hasattr(i, "description") ... line as the culprit part in your highlighted sentence from your first screenshot is i['description']. This always assumes an attack i will have a key (or entry) named description. If it doesn't, Python throws a KeyError. To safely retrieve an optional field like description:

i['description'] becomes i.get('description', 'No description provided') as it will first attempt to retrieve the description but if it fails, it will use the 'no description' string.

Hope that helps!

MarkDavidson commented 3 years ago

Hello @bobsyourmom and thank you for the bug report. TRAM has moved to https://github.com/center-for-threat-informed-defense/tram and the bug has been fixed in that repository so I am closing this issue. Thank you!

mitre-attack / tram

tram fails loading on the taxii feeds #59

Running tram.py fails once it gets to loading taxii feeds