( Ruby Client ) gRPC Proxy - GRPC::Cancelled: 1:unknown cause

[davissp14]

There was an issue that cropped up today that's preventing users from using LeaseKeepAliveRequest when connecting to through gRPC proxy using the Ruby etcdv3 client. https://github.com/davissp14/etcdv3-ruby

Details Etcd Version: 3.3.3

Setup: Single local Etcd node with single local grpc proxy.

Steps to reproduce:

1. Start up a local Etcd Node and gRPC Proxy

2. Install etcdv3 gem

   gem install etcdv3

3. irb

First path to error:

require 'etcdv3'

grpc_port = 23790

conn = Etcdv3.new(endpoints: "http://127.0.0.1:#{grpc_port}")
lease_id = conn.lease_grant(1000)["ID"]
conn.lease_keep_alive_once(lease_id)

GRPC::Cancelled: 1:unknown cause
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:26:in `check_status'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:209:in `block in read_loop'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `loop'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `read_loop'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/etcdv3-0.8.3/lib/etcdv3/lease.rb:26:in `each'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/etcdv3-0.8.3/lib/etcdv3/lease.rb:26:in `lease_keep_alive_once'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/etcdv3-0.8.3/lib/etcdv3/connection.rb:23:in `call'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/etcdv3-0.8.3/lib/etcdv3/connection_wrapper.rb:14:in `handle'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/etcdv3-0.8.3/lib/etcdv3.rb:121:in `lease_keep_alive_once'
    from (irb):5
    from /Users/shaun/.rbenv/versions/2.2.4/bin/irb:11:in `<main>'

Second path to error.

require 'etcdv3'

grpc_port = 23790

conn = Etcdv3.new(endpoints: "http://127.0.0.1:#{grpc_port}")
lease_id = conn.lease_grant(1000)["ID"]

stub = Etcdserverpb::Lease::Stub.new("127.0.0.1:#{grpc_port}", :this_channel_is_insecure)
request = Etcdserverpb::LeaseKeepAliveRequest.new(ID: lease_id)
stub.lease_keep_alive([request], metadata: {}).each do |resp| 
 return resp
end 

GRPC::Cancelled: 1:unknown cause
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:26:in `check_status'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:209:in `block in read_loop'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `loop'
    from /Users/shaun/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/grpc-1.12.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `read_loop'
    from (irb):80:in `each'
    from (irb):80
    from /Users/shaun/.rbenv/versions/2.2.4/bin/irb:11:in `<main>'

More information can be found here: https://github.com/davissp14/etcdv3-ruby/pull/117

[gyuho]

Hmm, are you using gRPC v1.12? etcd still depends on gRPC v1.7.x. We plan to upgrade to v1.12 though (for v3.4).

[davissp14]

Nah, I was just testing gRPC 1.12 to see if it made a difference. The gem is currently pinned at 1.6.7.

Holding off on unpinning 1.6.7 until this gets resolved: https://github.com/grpc/grpc/issues/15314

[gyuho]

I see.

So, this doesn't happen in v3.2 gRPC proxy?

[davissp14]

I just tested 3.2.6 and the issue is still present.

[gyuho]

Hmm, did lease keep alive once API ever work with Ruby client? We haven't changed anything in lease API, for awhile.

[davissp14]

I've never used the LeaseKeepAlive feature with the gRPC Proxy, so I can't say for sure. There's no issues with the Lease functionality when we target a standard member node, this issue only crops up when someone is connecting through the gRPC proxy.

https://github.com/davissp14/etcdv3-ruby/pull/117#issue-189497925

[davissp14]

Any updates?

cc:// @gyuho

[daMupfel]

This also happens when using grpc proxy and the https://github.com/kragniz/python-etcd3 python client. Refresh on leases are not working. Are there any leads what the problem is? Is this a client problem or a proxy problem?

[Sheph]

Environment: python: 2.7 python-etcd3: 0.10.0 etcd/etcd proxy: 3.3.9

the same problem. any progress on this ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.