Add MISP (Malware Information Sharing Platform) integration

[elhoim]

MISP is a platform to exchange IOCs. It would be great to have integration with it, more specifically to lookup IOCs in MISP. Or implement it the other way around, download the IOCs from MISP, and then compare those to the ones from CRITS.

MISP has a key-authenticated REST API available. Some details and a reference implementation in python.

Existing MISP data types that might be of interest to be used: yara rules, ip, hostname, url, filename, mutex, sha1, md5, sha256.

New data types to be implemented in the future: URI-regexp, filename-regexp, ssdeep, imphash, pehash, authentihash, pipe, mailslot, mutex, SSL certificates attributes, windows service name.

I can arrange access to a MISP instance hosting IOCs if you need it for testing.

[TheDr1ver]

I've created a tabbed service that allows you to take indicators related to a TLO and upload them to MISP via the API, but it doesn't allow for searching MISP IOCs or importing MISP IOCs into CRITs like you suggested.

Unfortunately, the code feels a bit messy for submitting a proper PR (and I don't really feel like doing a "cleaner" re-write from the ground up at the moment), but it works, so feel free to try it out - https://github.com/TheDr1ver/crits_services/tree/misp_service2/misp_service

[SecurityPhoton]

Is the misp tabbed service is working with current version of crits? I've managed to run it with pymisp==2.4.68 (not latest) and django==1.8.18 but still have some problem - could not add any objects on tab - gives error "Error adding relationship: Could not find Sample" when clicking + sign. Any ideas on this?

[TheDr1ver]

Unfortunately, I haven't had a chance to update my CRITs instance in a bit, so I haven't checked if the tabbed service works with the current version.

If you can post your full error log I might be able to figure out why it's not working for you when I have a chance to dig into it.

[SecurityPhoton]

Actually there is no errors in log. When you click the plus sign the URL in server log appear with status code 200. [23/Apr/2018 16:03:22] "POST /indicators/from_obj/ HTTP/1.1" 200 81 The text of error is balloon hint when you mouse over the plus sign. It looks like JavaScript problem. I will provide a screen tomorrow. I can also provide you an errors when using django version 1.11.12 (that is installed with bootstrap and have some issues with url libs and needs new HTML template). Also do i need a python ver 3 for this service?

[apolkosnik]

It looks like it might need a few small changes to run properly on the latest Django

[SecurityPhoton]

This the error that I was talking previously (using centos 7, pymisp==2.4.68 (not latest) and django==1.8.18 ).

When using django 1.11.12 the error log looks like: `[root@cent-crits ~]# cd /data/crits/ [root@cent-crits crits]# ./script/server Attempting to start runserver on port 8080 Performing system checks...

Unhandled exception in thread started by <function wrapper at 0x5891848> Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/django/utils/autoreload.py", line 228, in wrapper fn(*args, kwargs) File "/usr/lib64/python2.7/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run self.check(display_num_errors=True) File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 359, in check include_deployment_checks=include_deployment_checks, File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks return checks.run_checks(kwargs) File "/usr/lib64/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks new_errors = check(app_configs=app_configs) File "/usr/lib64/python2.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config return check_resolver(resolver) File "/usr/lib64/python2.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver return check_method() File "/usr/lib64/python2.7/site-packages/django/urls/resolvers.py", line 254, in check for pattern in self.url_patterns: File "/usr/lib64/python2.7/site-packages/django/utils/functional.py", line 35, in get res = instance.dict[self.name] = self.func(instance) File "/usr/lib64/python2.7/site-packages/django/urls/resolvers.py", line 405, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/usr/lib64/python2.7/site-packages/django/utils/functional.py", line 35, in get res = instance.dict[self.name] = self.func(instance) File "/usr/lib64/python2.7/site-packages/django/urls/resolvers.py", line 398, in urlconf_module return import_module(self.urlconf_name) File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module import(name) File "/data/crits/crits/urls.py", line 30, in url(r'^services/', include('crits.services.urls')), # Services File "/usr/lib64/python2.7/site-packages/django/conf/urls/init.py", line 50, in include urlconf_module = import_module(urlconf_module) File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module import(name) File "/data/crits/crits/services/urls.py", line 31, in url(r'^%s/' % d, include('%s.urls' % d))) File "/usr/lib64/python2.7/site-packages/django/conf/urls/init.py", line 50, in include urlconf_module = import_module(urlconf_module) File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module import(name) File "/data/services/misp_service/urls.py", line 1, in from django.conf.urls import patterns ImportError: cannot import name patterns`

I actually tried to fix this rewriting a file https://github.com/pontarr/misp_service/blob/master/urls.py and partially this file https://github.com/pontarr/misp_service/tree/master/templates and this helped to run the service without errors but it is not displaying in crits interface. I'm not very familiar with the python and django, I just need to run a bunch of crits and misp service.

`[root@cent-crits ~]# cd /data/crits [root@cent-crits crits]# ./script/server Attempting to start runserver on port 8080 Performing system checks...

/usr/lib/python2.7/site-packages/pymisp/api.py:20: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4 warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4") /usr/lib/python2.7/site-packages/pymisp/mispevent.py:45: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4 warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4") /usr/lib/python2.7/site-packages/pymisp/api.py:39: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4 warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4") System check identified no issues (0 silenced). April 24, 2018 - 06:17:38 Django version 1.11.12, using settings 'crits.settings' Starting development server at http://0.0.0.0:8080/ Quit the server with CONTROL-C.

nginx: [pid: 5681|app: 0|req: 9/9] 192.168.16.245 () {52 vars in 1755 bytes} [Tue Apr 24 07:25:32 2018] POST /services/misp_service/Sample/5aa91d61ad951d2a41102533/ => generated 29568 bytes in 87 msecs (HTTP/1.1 500) 6 headers in 196 bytes (1 switches on core 0)

webtrace: Environment:

Request Method: GET Request URL: http://192.168.22.106:8080/services/misp_service/Sample/5aa91d61ad951d2a41102533/

Django Version: 1.11.12 Python Version: 2.7.5 Installed Applications: ('crits.core', 'crits.dashboards', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.messages', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.staticfiles', 'crits.actors', 'crits.campaigns', 'crits.certificates', 'crits.domains', 'crits.emails', 'crits.events', 'crits.indicators', 'crits.ips', 'crits.locations', 'crits.objects', 'crits.pcaps', 'crits.raw_data', 'crits.relationships', 'crits.samples', 'crits.screenshots', 'crits.services', 'crits.signatures', 'crits.stats', 'crits.targets', 'tastypie', 'tastypie_mongoengine', 'django_mongoengine', 'django_mongoengine.mongo_auth') Installed Middleware: ('django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware')

Traceback:

File "/usr/lib64/python2.7/site-packages/django/core/handlers/exception.py" in inner

41. response = get_response(request)

File "/usr/lib64/python2.7/site-packages/django/core/handlers/base.py" in _get_response

187. response = self.process_exception_by_middleware(e, request)

File "/usr/lib64/python2.7/site-packages/django/core/handlers/base.py" in _get_response

185. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/lib64/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view

23. return view_func(request, *args, **kwargs)

File "/data/services/misp_service/views.py" in get_relationships

20. types)

File "/data/services/misp_service/handlers.py" in gather_relationships

417. n['tlo_labels']=tlo_labels

Exception Type: UnboundLocalError at /services/misp_service/Sample/5aa91d61ad951d2a41102533/ Exception Value: local variable 'n' referenced before assignment `

[TheDr1ver]

I based most of the code off of the relationships service, so I'd suggest mimicking any changes that have been made to that service since I first uploaded the misp service in order to get it to work.

Looking at your repo compared to the changes to the relationship service, it seems like you're on the right track.

One thing I noticed is that your handlers.py doesn't seem to have been updated. Try updating that that and see if it changes anything.

https://github.com/crits/crits_services/commit/1198238d9acfb05d18ee3205ab16bcd19d3f3057#diff-987f8a2a36671dff6452e57ed8669e47

(see the changes there to relationships_service/handlers.py)

[SecurityPhoton]

Thanks for pointing to update handlers. Now its working with current django version but I still have those error "Error adding relationship: Could not find Sample". I searched the code and found 6 places where this error can be generated but I dont know what to do with this. I suspect that the error probably somewhere in a configuration or in operation with a DB but I do not know as it to find.

[aemon]

Can somebody help us to change existing service which transfer data from CRITS ti MISP? It will be commercial work.