kontena / kontena

The developer friendly container and micro services platform. Works on any cloud, easy to setup, simple to use.
https://www.kontena.io/
Apache License 2.0
1.47k stars 127 forks source link

[1.3] node etcd_health API breaks on RPC symbol keys #2356

Closed SpComb closed 7 years ago

SpComb commented 7 years ago

The /v1/nodes/:grid/:node/health API's {"etcd_health": ...} field is broken, and always returns null, causing the CLI to consider the node as unhealthy:

[API Client Request]: GET http://localhost:9292/v1/nodes/development/core-01/health | Headers: {Accept: application/json, Authorization: Bearer} 
[API Client Response]: Headers: {Content-Type: application/json}  | Status: 200 | Body: 
{"id":"XI4K:NPOL:EQJ4:S4V7:EN3B:DHC5:KZJD:F3U2:PCAN:46EV:IO4A:63S5","name":"core-01","node_number":1,"initial_member":true,"connected":true,"etcd_health":{"health":null,"error":null}}
⊗ Node core-01 is unhealthy

This seems to be because the JSON view uses symbol keys:

json.etcd_health do
  json.health @etcd_health[:health]
  json.error @etcd_health[:error]
end

But the RPC response has string keys after passing through MessagePack, both on the websocket and MongoPubsub layers:

D, [2017-05-23T12:42:15.204584 #19846] DEBUG -- WebsocketBackend: RPC response: [1, 1129066692, nil, {"health"=>true}]
D, [2017-05-23T12:42:15.220536 #19846] DEBUG -- V1::NodesApi: etcd_health={"health"=>true}

The agent sends the RPC response with symbol keys:

D, [2017-05-23T09:41:12.642537 #1] DEBUG -- Kontena::RpcServer: rpc response: Kontena::Rpc::EtcdApi#health {:health=>true}
SpComb commented 7 years ago

This is because the Kontena 1.2 MongoPubsub stored the RPC payload as a BSON::Document, which also normalize keys to strings, as in MessagePack, but the resulting BSON::Document from MongoPubsub#subscribe also acts as a HashWithIndifferentAccess, normalizing Symbol key lookups to Strings.

{ "_id" : ObjectId("592418009b0404a8b0cf4b73"), "channel" : "rpc_client", "data" : { "type" : "request", "id" : "5UNJ:XBVI:2MKI:EGDV:GY6L:J3KR:AWXR:BKRZ:XJS2:GPHN:7UVT:KPFD", "message" : [ 0, 565594856, "/etcd/health", [ ] ] }, "created_at" : ISODate("2017-05-23T11:07:44.513Z") }
{ "_id" : ObjectId("592418009b0404a8b0cf4b74"), "channel" : "rpc_client:565594856", "data" : { "message" : [ 1, 565594856, null, { "health" : true } ] }, "created_at" : ISODate("2017-05-23T11:07:44.773Z") }
api_1      | D, [2017-05-23T11:13:50.304796 #12] DEBUG -- V1::NodesApi: etcd_health={"health"=>true} class=BSON::Document @etcd_health[:health]=true