Connecting to internet services (passivetotal, virustotal, ...) through a proxy

[keram79]

Our MISP is placed in an internal environment, internet services are only reachable by going through a squid proxy (without any auth, as our MISP IP is whitelisted). While syncing with other MISP instances in the internet works after configuring the proxy in the "Settings-->Proxy" tab, misp-modules like virustotal, countrycode etc. fail. Is this not implemented yet, or did I just miss the required step while working through the readme?

www-data@misp:/usr/local/src/misp-modules$ /usr/local/bin/misp-modules
HTTPConnectionPool(host='www.geognos.com', port=80): Max retries exceeded with url: /api/en/countries/info/all.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f35addd2438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
2016-10-05 14:47:25,682 - misp-modules - INFO - Launch MISP modules server from current directory.
2016-10-05 14:47:25,682 - misp-modules - INFO - Helpers loaded cache.py
2016-10-05 14:47:25,686 - misp-modules - INFO - MISP modules ocr imported
2016-10-05 14:47:25,686 - misp-modules - INFO - MISP modules testimport imported
2016-10-05 14:47:25,793 - misp-modules - INFO - MISP modules stiximport imported
2016-10-05 14:47:25,795 - misp-modules - WARNING - MISP modules countrycode failed due to HTTPConnectionPool(host='www.geognos.com', port=80): Max retries exceeded with url: /api/en/countries/info/all.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f35aaf8eb38>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[Rafiot]

aha, good point, the proxy parameters aren't passed to the modules.

A quick fix would be to start misp-modules with http_proxy set but it would be much better to pass the proxy parameters from the MISP config to the modules.

[ScottyAU]

@Rafiot can you elaborate on starting with http_proxy as an interim fix?

I just tried that but couldn't get it to work - most likley user error - but just wanted to check i am doing it correctly...

I've triple checked and the http_proxy and https_proxy vars are set on the box for all users...

Thanks!

[frennkie]

How is this issue related to #61 (duplicate?!)?

Where would be the best place to put the configuration of the proxy? I see two options: 1) config file somewhere in the file system which is read and interpreted by tornado 2) in the main MISP database - being passed to tornado when a module is called

I guess 1) is easier to implement and 2) is more user friendly...

I don't have any experience with Cake/PHP but I could look into implementing it independently in Python with a config file for the misp-modules.

[typonino]

I have tested two things:

• script using "requests" lib directly for web access
• script using specific lib which finally use "requests" lib

I have created a "modules.cfg" file in the expansion module folder with something like this:

[PROXY]
http = http://user:pass@proxy.gw.com:80
https = http://user:pass@proxy.gw.com:80
[SSL]
cafile = /path/to/my/ca

then for the first case add something like this:

import configparser
import os

session = requests.Session()

try:
    config = configparser.ConfigParser()
    config.read(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'modules.cfg'))
    if 'PROXY' in config.sections():
        proxies = {}
    if 'http' in config['PROXY']:
        proxies['http'] = config.get('PROXY', 'http')
    if 'https' in config['PROXY']:
        proxies['https'] = config.get('PROXY', 'https')
        if 'http' in proxies or 'https' in proxies:
        session.proxies = proxies
    if 'SSL' in config.sections()
        if 'cafile' in config['SSL']:
        session.verify = config.get('SSL', 'cafile')
except:
    pass

then use the "session" object instead of "requests" in code.

for second case add something like this:

import configparser
import os

try:
    config = configparser.ConfigParser()
    config.read(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'modules.cfg'))
    if 'PROXY' in config.sections():
        if 'http' in config['PROXY']:
            os.environ["HTTP_PROXY"] = config.get('PROXY', 'http')
        if 'https' in config['PROXY']:
            os.environ["HTTPS_PROXY"] = config.get('PROXY', 'https')
    if 'SSL' in config.sections():
        if 'cafile' in config['SSL']:
            os.environ["REQUESTS_CA_BUNDLE"] = config.get('SSL', 'cafile')
except:
    pass

the second way, simulates passing the environment variable to the script and requires no further modification and if exception occurred, script still continues with no matter.

Regards,

[xg5-simon]

Is this feature likely to be added as a milestone?

@ScottyAU did you eventually have any luck starting the misp-modules with http_proxy and https_proxy environmental variables set?

[ScottyAU]

@xg5-simon we did with some. Did you have a problem with one in particular? Happy to check ours to see.

[chrisinmtown]

We also need to configure our misp-modules installation for a proxy. I hope that you all found a good solution in the 3 years that this issue has been open. Will you please share?

p.s. to @frennkie I do not think this is dupe of #61, that calls for per-module proxies, and I think this issue is just about a global proxy setting for the misp-modules feature.