[flaky test] systemtest TestRUMXForwardedFor occasionally fails

[simitt]

The systemtest TestRUMXForwardedFor occasionally fails when resovling the geo information. This might happen when the tests are run on a different instance. If I remember correctly, @stuartnelson3 did some first analysis on this a while back.

[2021-10-04T09:14:16.190Z] === RUN   TestRUMXForwardedFor

[2021-10-04T09:14:16.190Z]     rum_test.go:62:   map[string]interface{}{

[2021-10-04T09:14:16.190Z]              "events": []interface{}{

[2021-10-04T09:14:16.190Z]                  map[string]interface{}{

[2021-10-04T09:14:16.190Z]                      "@timestamp": string("dynamic"),

[2021-10-04T09:14:16.190Z]                      "agent":      map[string]interface{}{"name": string("rum-js"), "version": string("5.5.0")},

[2021-10-04T09:14:16.190Z]                      "client": map[string]interface{}{

[2021-10-04T09:14:16.190Z]                          "geo": map[string]interface{}{

[2021-10-04T09:14:16.190Z]                              ... // 2 identical entries

[2021-10-04T09:14:16.190Z]                              "country_iso_code": string("AU"),

[2021-10-04T09:14:16.190Z]                              "country_name":     string("Australia"),

[2021-10-04T09:14:16.190Z]                              "location": map[string]interface{}{

[2021-10-04T09:14:16.190Z]         -                        "lat": float64(-31.9474),

[2021-10-04T09:14:16.190Z]         +                        "lat": float64(-31.9674),

[2021-10-04T09:14:16.190Z]         -                        "lon": float64(115.8648),

[2021-10-04T09:14:16.190Z]         +                        "lon": float64(115.8621),

[2021-10-04T09:14:16.190Z]                              },

[2021-10-04T09:14:16.190Z]                              "region_iso_code": string("AU-WA"),

[2021-10-04T09:14:16.190Z]                              "region_name":     string("Western Australia"),

[2021-10-04T09:14:16.190Z]                          },

[2021-10-04T09:14:16.190Z]                          "ip": string("220.244.41.16"),

[2021-10-04T09:14:16.190Z]                      },

[2021-10-04T09:14:16.190Z]                      "ecs":   map[string]interface{}{"version": string("dynamic")},

[2021-10-04T09:14:16.190Z]                      "event": map[string]interface{}{"ingested": string("dynamic"), "outcome": string("unknown")},

[2021-10-04T09:14:16.190Z]                      ... // 8 identical entries

[2021-10-04T09:14:16.190Z]                  },

[2021-10-04T09:14:16.190Z]              },

[2021-10-04T09:14:16.190Z]           }

[2021-10-04T09:14:16.190Z]         

[2021-10-04T09:14:16.190Z]         Run `make update check-approvals` to verify the diff

[axw]

I wonder if this is related to Elasticsearch automatically updating its MaxMind DB. In that case we might find another location which maps with less error, or otherwise just ignore the specific lat/lon when diffing.

[marclop]

I can take a look, Would be reasonable to not assert the exact values just they are set and aren't zero? Having the exact value assertion seems out of the scope of the test since the apm-server isn't doing the geolocation, right?

[axw]

@marclop I think that would be reasonable. We currently use approvals for this which might make it tricky, but we probably don't need to; we could just check that the X-Forwarded-For header value was threaded through and picked up by the geoIP processor.