search

awslabs / amazon-kinesis-client-ruby

A Ruby interface for the Amazon Kinesis Client Library. Allows developers to easily create robust application to process Amazon Kinesis streams in Ruby.
Apache License 2.0
146 stars 55 forks source link

Graceful Shutdown of Workers #42

Open cbatson opened 3 years ago

cbatson commented 3 years ago

Similar to https://github.com/awslabs/amazon-kinesis-client-nodejs/issues/34

Would like to see this functionality in the Ruby client for MultiLangDaemon.

Thank you.

devshorts commented 4 months ago

@pfifer shutdown doesn't work at all. https://github.com/awslabs/amazon-kinesis-client/pull/174 its easily reproducible with the given samples as well. Just start it up, hit control+c, nothing ever gracefully exits it just spews errors and blows up.

The ioproxy class is getting the signal propagated to it, it feels like signal handling needs to be caught by the java process and not sent down to the child process until after the shutdown initiate has occurred.

2024-05-09 16:48:55,991 [ShardRecordProcessor-0001] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream kclrbsample: listing shards with list shards request ListShardsRequest(StreamName=kclrbsample)
2024-05-09 16:48:56,064 [ShardRecordProcessor-0001] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing ProcessRecordsMessage to child process for shard shardId-000000000000
^C2024-05-09 16:48:56,070 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [/Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:31:in `readline': Interrupt] for shard shardId-000000000000
/Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:31:in `readline': Interrupt
2024-05-09 16:48:56,070 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:31:in `read_line'] for shard shardId-000000000000
    from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:31:in `read_line'
2024-05-09 16:48:56,070 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:45:in `read_action'] for shard shardId-000000000000
    from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/io_proxy.rb:45:in `read_action'
2024-05-09 16:48:56,070 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/kcl_process.rb:42:in `run'] for shard shardId-000000000000
    from /Users/devshorts/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/aws-kclrb-2.1.1/lib/aws/kclrb/kcl_process.rb:42:in `run'
2024-05-09 16:48:56,070 [multi-lang-daemon-0001] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [  from /Users/devshorts/src/external/amazon-kinesis-client-ruby/samples/sample_kcl.rb:99:in `<main>'] for shard shardId-000000000000
    from /Users/devshorts/src/external/amazon-kinesis-client-ruby/samples/sample_kcl.rb:99:in `<main>'
2024-05-09 16:48:56,070 [Thread-1] INFO  s.a.k.multilang.MultiLangDaemon [NONE] - Process terminated, will initiate shutdown.
2024-05-09 16:48:56,071 [multi-lang-daemon-0001] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Stopping: Reading STDERR for shardId-000000000000