search

MISP / MISP

MISP (core software) - Open Source Threat Intelligence and Sharing Platform
https://www.misp-project.org/
GNU Affero General Public License v3.0
5.3k stars 1.38k forks source link

Bug: Searches (UI or restSearch) for IPv6 addresses does not compress IPv6 addresses, resulting in "no results" for expanded addresses. #9042

Closed teward closed 1 year ago

teward commented 1 year ago

Expected behavior

The attributeSearch would return attributes matching the attribute value for an IPv6 address being queried.

Actual behavior

All attempts to find the data in API search or GUI search fail, suggesting that the system is unable to actually find expanded IPv6 addresses. This is due to the system compressing the IPv6 addresses and abbreviating them when storing them in the system. Search does not take an expanded IPv6 address and compress it, which results in a matching compressed IPv6 address attribute that would fit the expanded IPv6 to erroneously be ignored and therefore "no results" being returned.

Steps to reproduce

Create an event, and add the following attribute:

(NOTE: This value is pulled from OSINT data on tor exit nodes, this is not a 'private' data value hence why it's included here)

value: "2a0b:f4c2:0001:0000:0000:0000:0000:0190" category: Network activity type: ip-src

Publish event and attribute.

Attempt to search via restSearch for Attributes matching the expanded value, you should get "no results". You should get results on the compressed IP address of 2a0b:f4c2:1::190

Version

2.4.170

Operating System

Ubuntu

Operating System version

20.04

PHP version

7.4

Browser

N/A

Browser version

N/A

Relevant log output

No response

Extra attachments

Additional data here:

As part of parsing raw data values into MISP events, we have IPv6 addresses which have some problems with searching for the events. In order to NOT have duplicate events, we are trying to pull the attribute in if it exists, and update the lastSeen with the latest observation / data insert time.

HOWEVER, we are unable to actually search for the data properly. When running a restSearch via the API endpoint of /attributes/restSearch in the API client, and using { "value": "2a0b:f4c2:0001:0000:0000:0000:0000:0190" , "returnFormat": "json" } to get JSON data back, we get zero results from the search mechanism.

It was discovered on April 30th after discussion with Gitter chat that this is due to IPv6 addresses being compressed when stored in MISP, which results in an expanded IP search failing.

Code of Conduct

teward commented 1 year ago

Thanks to the Gitter chat, we identified that the issue is related to how MISP stores the data. MISP shortens/abbreviates the addresses but the restSearch system does not automatically shorten the IP address search so what is stored as 2a0b:f4c2:1::190 in the system does not show up with the expanded IPv6 address search.

This might be an issue that needs addressed in restSearch to autoshorten if it's an IP address.

However, on the coding side, now that we've established that MISP automatically shortens the IP address when storing it in the attributes, I can do this by making the project in Python and PyMISP instead of the scripting language chosen which directly interacts with the API with HTTP requests.

iglocska commented 1 year ago

Nice catch, definitely needs to be fixed asap.

iglocska commented 1 year ago

Fixed on develop, feel free to test it.