search

logstash-plugins / logstash-output-google_cloud_storage

Apache License 2.0
9 stars 26 forks source link

Cannot load key file #12

Closed pdehlke closed 7 years ago

pdehlke commented 7 years ago

Using any of logstash versions 2.3.4, 2.4.0, or 5.3.2, with this configuration:

input { file { path => "/opt/logs/trx.log" start_position => "beginning" } }

output { google_cloud_storage { bucket => "xxxxxx-prod-trx-logs" key_path => "/root/logstash-5.3.1/config/trxlogger.p12" service_account => "trxlogging@xxxxxx.iam.gserviceaccount.com" temp_directory => "/tmp/logstash-gcs" log_file_prefix => "logstash_gcs" max_file_size_kbytes => 1024 output_format => "plain" date_pattern => "%Y-%m-%dT%H:00" flush_interval_secs => 2 gzip => true uploader_interval_secs => 60 } }

Logstash refuses to start as it is unable to initialize the GCS output plugin:

[2017-04-27T20:26:34,289][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x31f1ecc3 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x46124a29 @metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2\"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x597465e3 @metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x165b5b3b @logger=#>, @strategy=#<LogStash::OutputDelegatorStrategies::Single:0x1923fcde @output=<LogStash::Outputs::GoogleCloudStorage bucket=>\"xxxxxx-prod-trx-logs\", key_path=>\"/root/logstash-5.3.1/config/trxlogger.p12\", service_account=>\"trxlogging@xxxxxx.iam.gserviceaccount.com\", temp_directory=>\"/tmp/logstash-gcs\", log_file_prefix=>\"logstash_gcs\", max_file_size_kbytes=>1024, output_format=>\"plain\", date_pattern=>\"%Y-%m-%dT%H:00\", flush_interval_secs=>2, gzip=>true, uploader_interval_secs=>60, id=>\"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_335e17aa-e6af-4c70-b0d9-de7b5e70d543\", enable_metric=>true, charset=>\"UTF-8\">, workers=>1, key_password=>\"notasecret\">, @mutex=#>, @id=\"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2\", @metric_events=#<LogStash::Instrument::NamespacedMetric:0x4c9bd091 @metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2\", :events]>, @output_class=LogStash::Outputs::GoogleCloudStorage>", :error=>"Invalid keyfile or passphrase"}

I've verified with openssl that /root/logstash-5.3.1/config/trxlogger.p12 is a valid key file and that I can use the passphrase notasecret to open it. auditd confirms that the file is being opened and read by logstash.

I've tried to use the workaround in #6 and trying to open the key file directly, with the same result:

irb(main):001:0> require "google/api_client" => true irb(main):002:0> Google::APIClient::KeyUtils.load_from_pkcs12('/root/logstash-5.3.1/config/trxlogger.p12', 'notasecret') ArgumentError: Invalid keyfile or passphrase from /root/logstash-2.3.4/vendor/jruby/lib/ruby/gems/shared/gems/google-api-client-0.8.6/lib/google/api_client/auth/key_utils.rb:88:in load_key' from /root/logstash-2.3.4/vendor/jruby/lib/ruby/gems/shared/gems/google-api-client-0.8.6/lib/google/api_client/auth/key_utils.rb:34:inload_from_pkcs12' from (irb):2:in evaluate' from org/jruby/RubyKernel.java:1079:ineval' from org/jruby/RubyKernel.java:1479:in loop' from org/jruby/RubyKernel.java:1242:incatch' from org/jruby/RubyKernel.java:1242:in catch' from /root/logstash-2.3.4/vendor/jruby/bin/irb:13:in(root)' irb(main):003:0>

Running on Ubuntu 16.04, java version "1.8.0_131", same behavior with logstash versions 2.3.4, 2.4.0, 5.3.1, and 5.3.2.

pdehlke commented 7 years ago

For the record:

This turns out to be java throwing java.security.InvalidKeyException: Illegal key size or default parameters and the whole gem-jruby-java chain swallowing it somewhere along the line. The solution is to install the JCE extensions: http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Perhaps that dependency could be documented in README.md?

Closing this issue.