elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
67 stars 3.5k forks source link

Logstash - customer wants a jdbc_streaming cache reused feature #8555

Open jkelastic opened 7 years ago

jkelastic commented 7 years ago

Customer Details - Use Case: Customer is currently using cache_expiration in jdbc_streaming. However, if the jdbc connection fails right after the cache_expiration the target => "query_results" will be blank.

For example:

cache_expiration: 10 seconds

Mysql connection failed at 20 seconds

Therefore, at 20 seconds the cache would expire and the mysql is severed. Customer would like to have a cache_reuse option.

Customer Details - Products/Versions: Logstash 5.5

Customer Details - Workarounds (if any): There are currently 3 workarounds but it's not exactly what customer wants.

Extend the cache_expiration to a high number. This is not sufficient as it means cache will stay longer and the latest jdbc_streaming data will not be fetched.

Have mysql database mirroring and proxy setup. If one database fails it will go to the other.

Pipe the select query output to a file and if there is a jdbc connection problem it will use the file data instead.

Customer Details - Feature Request: Customer would like a cache_reuse : true feature. If the jdbc connection isn't establish it will continue to use the last cache and ignore cache_expiration.

jkelastic commented 7 years ago

https://github.com/elastic/enhancements/issues/3135