search

MISP / misp-modules

Modules for expansion services, enrichment, import and export in MISP and other tools.
http://misp.github.io/misp-modules
GNU Affero General Public License v3.0
345 stars 234 forks source link

Weird networking and python permissions #131

Closed elreydetoda closed 7 years ago

elreydetoda commented 7 years ago

Hi,

I first wanted to share some information with you all in case you hadn't heard about it yet/if another user runs into the same problem.

INFORMATION

So I just wanted to let all know about this, and it makes sense but I ran into it on another persons system while working on it. If you are trying to run misp-modules on Ubuntu (or at least on NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" kernel version 4.4.0-83-generic) and you disable ipv6 but the device is still capable of ipv6 (shown from the netstat below for ssh). Then python will still try to default to ipv6, and since there is no ipv6 address you aren't able to start any of the modules...

tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1511/mysqld
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1327/redis-server 1
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1484/apache2
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1214/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1784/master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1484/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      1214/sshd

It shows here that weird things still happen in the kernel for ubuntu to allow this, but since there is no ipv6 address for any interface (including lo) then when trying to run misp-modules it will show the following

2017-07-18 12:05:15,782 - misp-modules - INFO - Launch MISP modules server from current directory.
2017-07-18 12:05:15,783 - misp-modules - INFO - Helpers loaded cache.py
2017-07-18 12:05:15,784 - misp-modules - INFO - MISP modules openiocimport imported
2017-07-18 12:05:15,784 - misp-modules - INFO - MISP modules stiximport imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules vmray_import imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules testimport imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules cuckooimport imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules email_import imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules ocr imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules mispjson imported
2017-07-18 12:05:15,787 - misp-modules - INFO - MISP modules passivetotal imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules asn_history imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules cve imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules reversedns imported
2017-07-18 12:05:16,005 - misp-modules - INFO - MISP modules countrycode imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules wiki imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules shodan imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules circl_passivedns imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules eupi imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules whois imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules xforceexchange imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules threatminer imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules sourcecache imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules iprep imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules circl_passivessl imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules vmray_submit imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules threatcrowd imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules ipasn imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules dns imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules otx imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules domaintools imported
2017-07-18 12:05:16,011 - misp-modules - INFO - MISP modules geoip_country imported
2017-07-18 12:05:16,011 - misp-modules - INFO - MISP modules virustotal imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules liteexport imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules cef_export imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules testexport imported
Traceback (most recent call last):
  File "/usr/local/bin/misp-modules", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/misp_modules/__init__.py", line 238, in main
    application.listen(port, address=listen)
  File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1943, in listen
    server.listen(port, address)
  File "/usr/local/lib/python3.5/dist-packages/tornado/tcpserver.py", line 142, in listen
    sockets = bind_sockets(port, address=address)
  File "/usr/local/lib/python3.5/dist-packages/tornado/netutil.py", line 197, in bind_sockets
    sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

The reason why (at least I believe from understanding what is going on and looking at the code while changing some things and then it working) is because of course it doesn't have an ipv6 address and when trying to bind to localhost it tries to bind to the localhost ipv6 address (non existent). So the way I fixed it was by going into the /usr/local/lib/python3.5/dist-packages/misp_modules/__init__.py and changing localhost to 127.0.0.1. after doing that everything worked properly from that standpoint.

QUESTION

I was curious if you have ever ran into that when trying to run misp-modules command, that no user was able to run it except for root? There is something wrong in being able to read the library or something for misp-modules and I don't know what is going on exactly... So I know it is a permission issue because when I try and run the command you recommend when running ubuntu to start the misp-modules sudo -u www-data misp-modules & or sudo -u www-data misp-modules -s & I get the following error

Traceback (most recent call last):
  File "/usr/local/bin/misp-modules", line 7, in <module>
    from misp_modules import main
ImportError: cannot import name 'main'

so I did some investigating and when I run python3 for an interactive shell (as an unprivileged user) and then do the following

import misp_modules
help(misp_modules)

I get the following

Help on package misp_modules:
NAME
    misp_modules
PACKAGE CONTENTS
FILE
    (built-in)
(END)

but when I do the same as root or sudoing this is what I get

Help on package misp_modules:

NAME
    misp_modules

DESCRIPTION
    # -*- coding: utf-8 -*-
    #
    # Core MISP expansion modules loader and web service
    #
    # Copyright (C) 2016 Alexandre Dulaunoy
    # Copyright (C) 2016 CIRCL - Computer Incident Response Center Luxembourg
    #
    # This program is free software: you can redistribute it and/or modify
    # it under the terms of the GNU Affero General Public License as published by
    # the Free Software Foundation, either version 3 of the License, or
    # (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU Affero General Public License for more details.
    #
    # You should have received a copy of the GNU Affero General Public License
    # along with this program.  If not, see <http://www.gnu.org/licenses/>.
...

which shows that it imports correctly, so do you all possibly know what it could be? I did the install properly and when I run sudo misp-modules everything works properly, but if I don't do a sudo or run as root then it won't be able to import everything ( as I showed you from the error above). Any ideas? Because I really don't want to run this as root but as the www-data user like how you all say.

thank you for this awesome product I am loving it so far!

krypto29s commented 7 years ago

Bump. Still having issues with Misp Modules will not run as any user other than root, and Stix/Cybox/MixBox shows the libraries are not installed on the web.

elreydetoda commented 7 years ago

@krypto29s figured it out, it was a weird permissions error that wasn't allowing everyone to access python libraries. So we ran sudo chmod -R ugo+rX /usr/local/lib/python3.5/dist-packages/ and after that we were able to execute misp-modules from any user.