logstash-plugins / logstash-input-salesforce

Logstash input for pulling objects from salesforce
Apache License 2.0
16 stars 29 forks source link

add feature to support retrieving changed records only. #24

Open brendanjlynch4014 opened 3 years ago

brendanjlynch4014 commented 3 years ago

This is request to add feature to support retrieving only changed records since the last execution. This may require the feature for scheduling to be completed also.

Since the salesforce SOQL query is extremely similar to JDBC SQL query, this plugin should be able to implement a similiar feature included in JDBC.

allow input fields to control some behavior like: use_column_value => true tracking_column => "LastModifiedDate" last_run_metadata_path => "/PATH TO FILE/PersistMyLastValue.yml" tracking_column_type=> "timestamp"

The plugin should incorporate the tracking column persisted value into the query string and append an Order By the tracking column when activated.

when the current query complete process the records it should write out a new peristed value based on the value of the last record.

Then the next query would pick up from the last processed message.