logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Tracking column in mongo input not working #383

Open DeNReLoaDeD opened 4 years ago

DeNReLoaDeD commented 4 years ago

Hello,

tracking_column doesn't work when query statement is against mongo. Debugging, i've seen that response in jdbc.rb comes like this:

{:document => { ... mongo data ...}}

In this method 'get_column_value', line:

  if !row.has_key?(@tracking_column.to_sym)

row.has_key only evaluates the first level of the hash, so all nested keys are omitted. Because of this, always is showing the message 'tracking_column not found in dataset.

I've edited the code, as a workarround, to also check if it's in the second level of the hash, and if it is, pick the value from there (second if (!row.dig...)):

public
def get_column_value(row)
  if !row.has_key?(@tracking_column.to_sym)
    if !row.dig(:document, @tracking_column).nil?
          @sql_last_value = row.dig(:document, @tracking_column)
    elsif !@tracking_column_warning_sent
      @logger.warn("tracking_column not found in dataset.", :tracking_column => @tracking_column)
      @tracking_column_warning_sent = true
    end
    # If we can't find the tracking column, return the current value in the ivar
    @sql_last_value
  else
    # Otherwise send the updated tracking column
    row[@tracking_column.to_sym]
  end
end

Now, it's working fine. I'm not a ruby developer and i'm pretty sure that there are better ways to do this, so if it could be fixed would be awesome.

Thanks!