IDN (Internationalized domain names) support

[romap0]

Host operating system: output of uname -a

Linux db6787aaa499 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter -version

0.16.0

What is the blackbox.yml module config.

default

What is the prometheus.yml scrape config.

global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://мтр24.рф #doesnt work
        - http://xn--24-7lcqj.xn--p1ai #works
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2020-05-16T22:20:32.688701072Z caller=main.go:304 module=http_2xx target=http://мтр24.рф level=info msg="Beginning probe" probe=http timeout_seconds=119.5
ts=2020-05-16T22:20:32.689183006Z caller=http.go:318 module=http_2xx target=http://мтр24.рф level=info msg="Resolving target address" ip_protocol=ip6
ts=2020-05-16T22:20:32.689332368Z caller=http.go:318 module=http_2xx target=http://мтр24.рф level=error msg="Resolution with IP protocol failed" err="lookup мтр24.рф: no such host"
ts=2020-05-16T22:20:32.689371761Z caller=main.go:119 module=http_2xx target=http://мтр24.рф level=error msg="Error resolving address" err="lookup мтр24.рф: no such host"
ts=2020-05-16T22:20:32.689399341Z caller=main.go:304 module=http_2xx target=http://мтр24.рф level=error msg="Probe failed" duration_seconds=0.000408558

What did you do that produced an error?

What did you expect to see?

probe_success 1

What did you see instead?

probe_success 0

[brian-brazil]

Hmm, how does normal Prometheus scraping deal with this? Does Go even support these?

[romap0]

I'm not familiar with Go, but as I tested with net/http module it works.

Code

```go package main import ( "fmt" "io/ioutil" "log" "net/http" ) func main() { url := "http://мтр24.рф" fmt.Println("URL:>", url) resp, err := http.Get(url) if err != nil { log.Fatal(err) } fmt.Println("response status:", resp.StatusCode) body, err := ioutil.ReadAll(resp.Body) fmt.Println("response body:", string(body)) } ```

Logs

``` URL:> http://мтр24.рф response status: 200 response body:

```

[brian-brazil]

Okay, so likely our own resolution stuff can't handle this.

[mem]

There's a third-party library for Go that deals with IDN.

The way we are dealing with it is by taking the IDN hostnames, applying the punycode transformation and passing that as the target to. For ping and DNS it's just a matter of applying punycode to the input. For TCP we have to split the hostname and the port. For HTTP we parse the URL, punycode the hostname and build a new URL. For DNS we are also transforming the query (so the BBE config has punycoded strings, not UTF8). Transforming regular hostnames using punycode yields back regular hostnames, so you don't have special cases.

The ugly part is that you see punycoded strings where you expect something else.

I could work on a PR to handle this inside BBE if we decide how to handle that case ("it's OK to see punycoded hostnames in logs", "I want to see UTF8 in logs", "...").

[brian-brazil]

I wouldn't worry about the logs, there's an existing PR #640 which is having issues with some of the finer details.