search

awslabs / amazon-kinesis-client-python

Amazon Kinesis Client Library for Python
Apache License 2.0
368 stars 223 forks source link

KinesisClientLibIOException Shard [shardId-000000000000] is not closed #66

Open jl-DaDar opened 6 years ago

jl-DaDar commented 6 years ago

Getting this error with Kinesalite. It was working fine the first time I ran the kcl python app, but once I tried to delete local kinesalite stream and local dynamodb table, i start getting this error below ever since. please help!

严重: Caught exception while sync'ing Kinesis shards and leases com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException: Shard [shardId-000000000000] is not closed. This can happen if we constructed the list of shards while a reshard operation was in progress. at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.assertClosedShardsAreCoveredOrAbsent(ShardSyncer.java:206) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.cleanupLeasesOfFinishedShards(ShardSyncer.java:652) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.syncShardLeases(ShardSyncer.java:141) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.checkAndCreateLeasesForNewShards(ShardSyncer.java:88) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask.call(ShardSyncTask.java:68) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:427) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:356) at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:114) at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:61) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748)

Belkhojayev commented 6 years ago

Unfortunately, I can't help you cause I'm still setting up KCL first time myself.

Воскресенье, 19 ноября 2017, 17:27 +01:00 от Ryan Cheng notifications@github.com:

Getting this error with Kinesalite. It was working fine the first time I ran the kcl python app, but once I tried to delete local kinesalite stream and local dynamodb table, i start getting this error below ever since. please help! 严重: Caught exception while sync'ing Kinesis shards and leases com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException: Shard [shardId-000000000000] is not closed. This can happen if we constructed the list of shards while a reshard operation was in progress. at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.assertClosedShardsAreCoveredOrAbsent(ShardSyncer.java:206) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.cleanupLeasesOfFinishedShards(ShardSyncer.java:652) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.syncShardLeases(ShardSyncer.java:141) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.checkAndCreateLeasesForNewShards(ShardSyncer.java:88) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask.call(ShardSyncTask.java:68) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:427) at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:356) at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:114) at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.call(MultiLangDaemon.java:61) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub , or mute the thread .

С уважением,

beldan@mail.ru

jl-DaDar commented 6 years ago

i seem to figure out the problem, but I cannot fix this. There is a problem with the python client on using the right region. I am using local dynamodb with cn-north-1 region in aws configure, but when a local end point (http://localhost:4567) is set, KCL uses us-east-1. As a result, when I run KCL, the KCL uses us-east-1 local dynamodb , while the rest of my code (that uses pynamodb) uses cn-north-1 database. It is causing a lot of strange issues. When I switch aws configure to us-east-1, I can run the code.

Can someone from aws please fix this?

jl-DaDar commented 6 years ago

so now the code will run but I keep on getting this exception...

十一月 20, 2017 6:56:44 上午 com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics 警告: Could not publish 18 datums to CloudWatch com.amazonaws.services.cloudwatch.model.AmazonCloudWatchException: The security token included in the request is invalid. (Service: AmazonCloudWatch; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: f805b31b-cdbf-11e7-b33c-fdfd5e1c3279) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1073) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1049) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:957) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:934) at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:144) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90) at java.lang.Thread.run(Thread.java:748)