wip: add socks5 support

thomasf commented 6 years ago

Work in progress for #9

basic configuration

I'm working towards the configuration style below. The ports: field is based on code for a docker-compose.yml ports:" parser from some docker package..


  myservice:
    method: http
    http:
      ports:
        - 8100-8120                  # myservice:8100 to myservice:8120
        - 192.168.1.1:8100-8120:8300 # myservice:8300 to myservice:8320

  myservice2:
    method: http
    http:
      port: 8100                     # myservice2:1 or myservice2:8100

easy/safe unique scrape host names

The config below would create the same scrape target instance names myservice:1 and myservice:2 for multiple actual hosts which isn't great and can lead to accidental mix ups in Prometheus queries.

 - job_name: 'myservice_host1'
    scrape_interval: 1s
    static_configs:
      - targets: ['myservice:1', 'myservice:2']
    scheme: https
    proxy_url: "socks5://host1:3000"
    metrics_path: /proxy
    tls_config:
        ...

 - job_name: 'myservice_host2'
    scrape_interval: 1s
    static_configs:
      - targets: ['myservice:1', 'myservice:2']
    scheme: https
    proxy_url: "socks5://host2:3000"
    metrics_path: /proxy
    tls_config:
        ...

While it is probably possible to avoid instance name collisions using relabeling inside Prometheus I think it's best to be able to ensure instance name uniqueness at the node level.

To solve this issue an optional -socks.prefix flag is introduced to exporter_exporter which enforces a FQN prefix to the virtual host names.

If an exporter_exporter instance is started with -socks.prefix cool.host1 the scrape target would then look like this:

 - job_name: 'myservice_host1'
    scrape_interval: 1s
    static_configs:
      - targets: ['cool.host1.myservice:1', 'cool.host1.myservice:2']
    scheme: https
    proxy_url: "socks5://host1:3000"
    metrics_path: /proxy
    tls_config:
        ...

To lessen the requirements of manual node configuration An additional boolean flag -socks.auto-prefix will try to set the prefix using os.Hostname() returns unless it returns an empty string or error in which case start up will be aborted by an fatal exit.

command line options

right now it looks like this (the descriptions will be revised later):

  -socks.listen-address string
        The address to listen on for accessing the metrics end points via made up internal host names.
  -socks.tls
        Use TLS with the socks proxy (default true)
  -socks.prefix
        ...
  -socks.auto-prefix (default False)
        ...

(QUESTION) can not use socks method + TLS + non-TLS at the same time

Because -socks.tls and -socks.listen-address are combined the socks method can not be enabled for both tls and non tls access simultaneously. I don't see it useful to have both working at the same time but we have to change some things if it should be supported.

(QUESTION) should it be allowed to configure the same service multiple times?

I'm a bit skeptical of this feature since it complicates the rules for what a valid configuration can be while not adding strong practical benefits.

This question can be deferred since it can be added later if needed

  myservice:
    method: http
    http:
      ports:
        - 8100-8120                  # myservice:8100 to myservice:8120
        - 192.168.1.1:8100-8120:8300 # myservice:8300 to myservice:8320

  myservice: # invalid config since port 8100 already was defined above
    method: http
    http:
      port: 8100

  myservice: # valid config since port 8400 isn't alloceted for myservice
    method: http
    http:
      ports:
      - "8101:8400"

For the directory based config myservice.app1.yml and myservice.app2.yml would work for allowing multiple instances.

(QUESTION) exposing multi port services for non socks methods

This question can be deferred since it can be added later if needed

I'm not sure if if is meaningful to implement the multiple ports per service scheme for the regular http and https serving options via URL parameters. The corresponding prometheus.yml would become very complicated. Could be useful if one wants to use a web browser to easily view the exporters but that isn't easy when using client certificates login anyway so maybe not a big win?

thomasf commented 6 years ago

@tcolgate could you comment on the user facing behavior stuff I wrote about in this issue description.. I would like to have most of it decided before I'm committing my time towards the underlying solution.

You can ignore any actual code for now, I'll inform you when it's ready for a discussion.