How to set port? Getting error.

rayjlinden commented 5 years ago

Trying to connect to kinesislite (via Localstack).

I get this error: Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Kinesis codec=><LogStash::Codecs::JSON id=>\"c4681ca6-d007-435d-8f2d-65f2aa01658a\", enable_metric=>true, charset=>\"UTF-8\">, id=>\"8cc337231c7e1d6704bc63e427fc031ffe14cafd60be3cebce74bc32007607f4\", region=>\"us-west-1\", additional_settings=>{\"kinesis_endpoint\"=>\"localstack\", \"kinesis_port\"=>4568, \"verify_certificate\"=>\"false\"}, kinesis_stream_name=>\"maxwell\", enable_metric=>true, application_name=>\"logstash\", checkpoint_interval_seconds=>60, initial_position_in_stream=>\"TRIM_HORIZON\">", :error=>"undefined method `with_kinesis_port' for #\nDid you mean? with_kinesis_endpoint", :thread=>"#"}

Here was my config for Logstash: input { kinesis { kinesis_stream_name => "maxwell" region => "us-west-1" codec => json { } additional_settings => { "kinesis_endpoint" => "localstack" "kinesis_port" => 4568 "verify_certificate" => false } } }

What am I doing wrong?

rayjlinden commented 5 years ago

Well I got further. My inout now looks like this:

input {
  kinesis {
    application_name => "maxwell"
    kinesis_stream_name => "maxwell"
    codec => json { }
    additional_settings => {
        "kinesis_endpoint" => "http://kinesislite:4568"
        "dynamo_db_endpoint" => "http://dynamodb:8000"
        "initial_lease_table_read_capacity" => 25
        "initial_lease_table_write_capacity" => 100
        "metrics_level" => "NONE"
    }
  }
}

However, I still get failures when it tries to run:

[2019-01-06T07:50:20,998][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Jan 06, 2019 7:50:21 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask call
SEVERE: Caught exception while sync'ing Kinesis shards and leases
com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1136)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1082)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
...

I also tried adding: echo "-Dcom.amazonaws.sdk.disableCertChecking" >> config/jvm.options

I saw elsewhere that helped with turning off cert checking on the java client.

In any case, I'm still stuck. Any ideas?

robbavey commented 5 years ago

@rayjlinden I just spun up a quick localstack docker instance on my local machine, and can successfully ingest from it by setting kinesis_endpoint and dynamodb_endpoint settings in the additional_settings block as you did above.

As this is a usage question, this question would be betterasked at https://discuss.elastic.co.

GitHub is for coding issues and error reporting.

logstash-plugins / logstash-input-kinesis

How to set port? Getting error. #61