mhart / kinesalite

An implementation of Amazon's Kinesis built on LevelDB
MIT License
808 stars 86 forks source link

Logstash kinesis-input-plugin with kinesis lite? #83

Open rayjlinden opened 5 years ago

rayjlinden commented 5 years ago

Has anyone got Logstash kinesis-input-plugin to work with kinesis lite?

I have this Logstash config like this:

input {
  kinesis {
    application_name => "maxwell"
    kinesis_stream_name => "maxwell"
    codec => json { }
    additional_settings => {
        "kinesis_endpoint" => "http://kinesislite:4568"
        "dynamo_db_endpoint" => "http://dynamodb:8000"
        "initial_lease_table_read_capacity" => 25
        "initial_lease_table_write_capacity" => 100
        "metrics_level" => "NONE"
    }
  }
}

I also do this for Logstash: echo "-Dcom.amazonaws.sdk.disableCertChecking" >> config/jvm.options to turn off ssl checking.

But I'm still getting this error:

WARNING: Received configuration for region as us-east-1.
Jan 06, 2019 7:50:20 AM com.amazonaws.http.AmazonHttpClient createSocketFactoryRegistry
WARNING: SSL Certificate checking for endpoints has been explicitly disabled.
[2019-01-06T07:50:20,998][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Jan 06, 2019 7:50:21 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask call
SEVERE: Caught exception while sync'ing Kinesis shards and leases
com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1136)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1082)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
...

I know this is not a kinesis lite topic per se. But I'm hoping someone here my have done this before.