multiple ip addresses when using x-forwarded_for

vukomir commented 4 years ago

Problem

If a request goes through multiple proxies, the IP addresses of each successive proxy is listed. This means, the right-most IP address is the IP address of the most recent proxy and the left-most IP address is the IP address of the originating client.

error dump an error event: error_class=GeoIP2::Error error="getaddrinfo failed: Name or service not known" location="/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-geoip-1.3.1/lib/fluent/plugin/filter_geoip.rb:175:in `lookup'" ...

Steps to replicate

X-Forwarded-For: 203.0.113.195, 70.41.3.18, 150.172.238.178

Expected Behavior

is there are multiple ip addresses in the filed that contains ip address it should use the first ip ...

Your environment

'fluent-plugin-concat' version '2.4.0' 'fluent-plugin-dedot_filter' version '1.0.0' 'fluent-plugin-detect-exceptions' version '0.0.13' 'fluent-plugin-elasticsearch' version '4.0.7' 'fluent-plugin-geoip' version '1.3.1' 'fluent-plugin-grok-parser' version '2.6.1' 'fluent-plugin-json-in-json-2' version '1.0.2' 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0' 'fluent-plugin-multi-format-parser' version '1.0.0' 'fluent-plugin-prometheus' version '1.6.1' 'fluent-plugin-record-modifier' version '2.0.1' 'fluent-plugin-rewrite' version '0.1.1' 'fluent-plugin-rewrite-tag-filter' version '2.2.0' 'fluent-plugin-systemd' version '1.0.2' 'fluentd' version '1.10.1'

okkez commented 4 years ago

You can apply filter record_transformer plugin before this plugin.

harshpandit007 commented 4 years ago

@vukomir : I'm having the similar issue with x_forwarded_for. did you figure it out using record_transformer?

harshpandit007 commented 4 years ago

With the help of @okkez and my friend @francisco-andrade, I was able to solve this. Just posting my solution here for future references:

#Grep first ip from x_forwarded_for list and put it into a new variable i.e. x_forwarded_for_ip
<filter nginx.access>
  @type record_transformer
  enable_ruby
  <record>
     x_forwarded_for_ip ${record["x_forwarded_for"].split(',')[0]}
  </record>
</filter>

# We use the above-created x_forwarded_for_ip in the geoip plugin as this expects only one IP in the input string.
<filter nginx.access>
  @type geoip
  backend_library geoip2_c
  # Set key name for the client ip address values
  geoip_lookup_keys     x_forwarded_for_ip
  # Specify key name for the country_code values
  <record>
    city ${city.names.en["x_forwarded_for_ip"]}
    latitude ${location.latitude["x_forwarded_for_ip"]}
    longitude ${location.longitude["x_forwarded_for_ip"]}
    country_code ${country.iso_code["x_forwarded_for_ip"]}
    country_name ${country.names.en["x_forwarded_for_ip"]}
    region_code ${subdivisions.0.iso_code["x_forwarded_for_ip"]}
    region_name ${subdivisions.0.names.en["x_forwarded_for_ip"]}
  </record>
</filter>

y-ken / fluent-plugin-geoip