Per module setting of requests options (proxies + verify)

[frennkie]

I ran into an issue with the settings that are used /can be provided to the modules for requests. On some modules I need to disable the HTTPS certificate verification while on others I don't want this security feature to be switched off.

I have a similar issue with the proxy settings for requests. Some modules query resources on the internet (e.g VT, Koodous) while others need to talk to systems on the corporate intranet (e.g. Cuckoo, a local MISP or Lastline instance). In the first case I need to use a proxy, but in the second configuration I have to avoid using the (internet) proxy.

I assume the proxy issue could be worked around with using the "no_proxy" environment setting.. although I don't like doing it that way.

So my suggestion would be to add two global configuration variables (one for proxy (proxies with default: {}) and one for HTTPS certificate verifyication (verify with default: True )?!)) which are then used as a default and which can be overridden on a per-module basis.

[Rafiot]

Yep, works for me. We may want to have an other option: client side certificate.

[frennkie]

@Rafiot would you put these settings into separate sections?

[proxy]
# http = http://user:pass@prx.example.com:3128
http =

[tls_verification]
# verify = False
verify = True

# ca_bundle_path = /root/my_ca_bundle.crt
ca_bundle_path =

[tls_client_certificate]
path = /root/my_client_cert.pem

Or just use a generic section like [http_client]

[Rafiot]

http_client section sounds good to me for now, but if it turns out we have much more options, multiple sections would be better.

[frennkie]

Not much more code yet, but the config would then look like this:

[http_client]
# http client settings for outgoing requests (e.g. download, VT)
# will be applied to requests and can be override per module

# Proxy specific settings:
# * https_proxy = e.g. http://user:pass@prx.example.com:3128 (will also set http_proxy)
# * no_proxy = host1.example.com,internal.domain
# If https_proxy and no_proxy are not set then settings from the environment will be used.
# If no proxy should be used then use https_proxy = False (overrides environment)
#https_proxy =
#no_proxy =

# tls_verify (default: True): Whether TLS certificates should be validated or not
#tls_verify = True|False

# tls_ca_bundle: Path to a (custom) ca/ca_bundle file for TLS certificate validation
#tls_ca_bundle = /root/my_ca_bundle.crt

# tls_client_cert: Path to a TLS client certificate (cert and key) which should be used 
# for authentication
#tls_client_cert = /root/my_client_cert.pem 

Then it would be possible to set/override e.g. tls_client_cert on a certain module (e.g. MISP).

Sounds sensible?

[Rafiot]

Looks good to me.