search

RedSiege / Just-Metadata

Just-Metadata is a tool that gathers and analyzes metadata about IP addresses. It attempts to find relationships between systems within a large dataset.
https://www.christophertruncer.com/just-metadata-intel-gathering-and-analysis-of-ip-metadata/
GNU General Public License v3.0
624 stars 96 forks source link

New IP feeds #33

Closed xme closed 7 years ago

xme commented 8 years ago

Hi Chris,

I just added some IP feeds...

ChrisTruncer commented 8 years ago

Just checked out the code, all looks great and runs well. The only thing is you're collecting the data for those extra feeds, but not doing anything with it at the moment. When you run the analysis module for the threat feeds, that data isn't coded into it so it's only analyzing the old threat feeds, and not the new ones you just added.

xme commented 8 years ago

Right! I'll add the code to the analysis module. In fact, I'm not using it. I export data and index them in Splunk...

/x

ChrisTruncer commented 8 years ago

Ha, that's funny. Glad to see a different use case for the data!

ChrisTruncer commented 8 years ago

Just to make you aware, I was waiting to review until the analysis module was added in to support this, but it also looks right now that there is a merge conflict, so I can't merge this in the current state.

xme commented 8 years ago

Hi Chris, I fixed the analysis module to support the new feeds. I'll check the config...

ChrisTruncer commented 8 years ago

Just tried running this, I'm getting a quick error:

Traceback (most recent call last):
  File "./JustMetadata.py", line 13, in <module>
    from common import orchestra
  File "/tmp/Just-Metadata/common/orchestra.py", line 58
    "ip_list" : "List IP addresses loaded in the framework",
              ^
SyntaxError: invalid syntax
xme commented 8 years ago

Fixed a stupid typo!

ChrisTruncer commented 8 years ago

Hey, I'm still running into problems. If I try to load IPs into the tool using the load command, it crashes.

Traceback (most recent call last): File "./JustMetadata.py", line 94, in <module> the_conductor.menu_system() File "/tmp/Just-Metadata/common/orchestra.py", line 270, in menu_system self.load_ips(self.user_command.split()[1]) File "/tmp/Just-Metadata/common/orchestra.py", line 189, in load_ips total_ssytems = add_ip(system) NameError: global name 'add_ip' is not defined

xme commented 8 years ago

Hi Chris, Bug fixed!

ChrisTruncer commented 8 years ago

Hey again,

I'm running into a bug when testing the ip_add and ip_delete command. I'm loading a file with 8 IPs in them. I then tried adding an IP which was already in the list, and it stored it as a separate object.

adding ip

duplicate ips

When I list now, it shows two "8.8.8.8" IPs. I'd ideally prefer for it to do one of two things:

  1. Have all the IPs uniqued and do not show duplicates
  2. Show all IPs, including duplicates (or more).

If I try adding 8.8.8.8 again (to make it have three 8.8.8.8 IPs loaded, it only shows two).

Also, if I try to remove the duplicate 8.8.8.8, it only shows one 8.8.8.8 (as expected). But if I try to remove the last 8.8.8.8 IP that's loaded in the framework, it says it's not loaded and won't delete it, but when listing the IP, it is still present.

ip list and delete

ip delete

xme commented 8 years ago

Hi Chris, Bug fixed, there was a remaining '\n'. Looks good now.

ChrisTruncer commented 7 years ago

Hey,

Sorry it took me so long to review this, I missed you updating it to address my question. So, I may have not explained well what I was looking for, and that's my fault. Just-Metadata should be allowed to contain duplicate IP addresses, it's definitely possible for the same IP to access "something" multiple times, and I think it's fine for Just-Metadata to have the duplicates. In fact, the way IP objects are laid out in the framework, Just-Metadata can easily account for duplicates of any IP address (https://github.com/ChrisTruncer/Just-Metadata/blob/master/common/orchestra.py#L150-L155). The main thing I was looking for during the ip_list is if there are duplicate IP addresses, they are represented in the list (so if 8 2.2.2.2 IPs, then it should show all 8), or it should uniq the list of IPs shown while maintaining the total number of instances of each IP in the backend.

From a usability perspective, I think it might make the most sense to show each IP just so users don't wonder why when they delete an IP, it might still be there (if there's 7 instances of an IP, and they delete one, there's six left, but it still only shows the uniqued IP). Thoughts?

ChrisTruncer commented 7 years ago

Also, if I remember correctly, the reason it didn't work correctly with the first submission is because a new IP object was created if adding an IP with ip_add even if the IP already was in Just-Metadata. It would probably be best if adding an IP (or removing one) that the loaded IP objects are searched to see if they contain the specific IP address being added or removed, and then adjust that accordingly.

ChrisTruncer commented 7 years ago

Closing this due to inactivity. This can always get added in later if so desired.

Thanks!