Support to accept dynamic connection string

kgaswin93 commented 7 years ago

Support In case of multiple applications connecting to same logstash and writing to different oracle servers. The connection string will come from the application to logstash which has to be used in your output plugin. Can you provide support for this, it is a basic use case which ideally should be covered.

Ex : output {
jdbc { driver_class => "oracle.jdbc.driver.OracleDriver" connection_string => "%{connection_string}" statement => [ "INSERT INTO table VALUES(?, "value" ]
} }

theangryangel commented 7 years ago

The recommendation I have is to use multiple jdbc outputs in your configuration, and then to use if statements to govern which output your event goes to. This is exactly how people have multiple elasticsearch outputs. If you need an example see https://gist.github.com/jhmartin/c09e450853e273ee88f2

Couple of issues with using a dynamic connection string:

The database connection pool is established before any events are sent. To acheive a dynamic connection string the plugin would have to be largely rewritten
It would make dealing with connection errors waaaaaaaaaaaaay more complicated than it is right now
I barely have the time to maintain the plugin as it is
To my knowledge no other logstash plugins support this behaviour, and instead recommend the multiple outputs in configuration solution

I'm honestly not likely to support this at all.

theangryangel commented 7 years ago

I'm happy to have a discussion if the multiple jdbc outputs doesn't suit you, but you're going to have to try really hard to persuade me.

kgaswin93 commented 7 years ago

Hi, My main issue is that I require some type of encryption on top of jdbc credentials. Clients do not accept without any encryption. So I thought, if it could accept dynamic fields, then I could write my own ruby code to decrypt and use it in your plugin. Please suggest a workaround or a solution.

Regards, Aswin

On 25-Jan-2017 6:05 PM, "Karl" notifications@github.com wrote:

I'm happy to have a discussion if the multiple jdbc outputs doesn't suit you, but you're going to have to try really hard to persuade me.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/theangryangel/logstash-output-jdbc/issues/72#issuecomment-275097402, or mute the thread https://github.com/notifications/unsubscribe-auth/AEnIwjVs1-oVUaYRUAP3YC_rMoXFu3lWks5rV0GKgaJpZM4LtayP .

theangryangel commented 7 years ago

My main issue is that I require some type of encryption on top of jdbc credentials

If you're concerned about credential loss, my personal opinion is that the logstash configuration file should be properly secured so that only your logstash user and root can access it. That solves a large bunch of problems immediately (I will admit not all, but many).

If that's not enough, in theory we could implement a patch in the plugin that allows credentials to be picked up from something like a java keystore. But you'd still have to unlock that some how.. Which likely means that the keystore password would be in the configuration file.. which means we're back to "if you can read the configuration you have the credentials". Which means it's not a great improvement, if any.

The only way around that would be some kind of hardware credential store, or maybe prompting for password on service start (which I personally don't think is practical).

The only other alternative I can think of, that you can use right now, is passing in credentials via environment variables. Which is possible in recent versions of logstash. Whether or not thats more secure is up for debate I suppose. Also they still wont be encrypted.

A couple of links that may be helpful on the subject of encrypted logstash configurations:

If you have a proposal that would work for you I'm willing to listen and consider implementing it.

theangryangel / logstash-output-jdbc

Support to accept dynamic connection string #72