search

davidkel / fabric-chaos-testing

Apache License 2.0
2 stars 1 forks source link

Perform manual testing on non-gateway peers going down cleanly in a submit scenario #28

Open davidkel opened 2 years ago

davidkel commented 2 years ago

The testing here needs to be split into 2 separate tasks

  1. test scenarios where we expect the system to continue to work because there are enough alternative peers to satisfy the used endorsement policy. We should try multiple different endorsement policies
  2. test scenarios where we expect the system to not be able to satisfy the endorsement policy. We need to see what kind of errors clients may receive in these scenarios

Endorsement policies to test

davidkel commented 2 years ago

Point 1.

Tested with endorsement policy of

  1. default (Majority)
  2. explicit (Majority): OR(AND(‘Org1MSP.member’,‘Org2MSP.member’),AND(‘Org1MSP.member’,‘Org3MSP.member’),AND(‘Org3MSP.member’,‘Org2MSP.member’))
  3. explicit (All): AND('Org1MSP.member','Org2MSP.member','Org3MSP.member')

issues raised

davidkel commented 2 years ago

Point 2.

Tested with explicit endorsement policy of all (Don't believe there is a need to test other endorsement policies)

I see the following different types of error

{"component":"CLIENT","timestamp":"2021-11-03T12:00:56.036Z","txnId":"7f41db59172999e592f27206e54e4573770be9e54522e1ff10684b0a328aa16a","stage":"Failed","message":"14 UNAVAILABLE: failed to select a set of endorsers that satisfy the endorsement policy Details: []"}
{"component":"CLIENT","timestamp":"2021-11-03T12:00:56.826Z","txnId":"9bbaefa6539d142c9252d670045c75535cfd8cc43ef4523940ac6159afe6053c","stage":"Failed","message":"10 ABORTED: failed to collect enough transaction endorsements, see attached details for more info Details: [{\"address\":\"peer0.org3.example.com:11051\",\"message\":\"rpc error: code = DeadlineExceeded desc = context deadline exceeded\",\"mspId\":\"Org3MSP\"}]"}

{"component":"CLIENT","timestamp":"2021-11-03T12:00:56.826Z","txnId":"8f10d89783898140f6927dc9d152bbc552dec815e572a82f980606c14c203c69","stage":"Failed","message":"10 ABORTED: failed to collect enough transaction endorsements, see attached details for more info Details: [{\"address\":\"peer0.org3.example.com:11051\",\"message\":\"rpc error: code = Unavailable desc = transport is closing\",\"mspId\":\"Org3MSP\"}]"}

{"component":"CLIENT","timestamp":"2021-11-03T13:31:19.214Z","txnId":"a9e0713159c7eeac3baed71dfb99f9f47b7bcceb25566685cd76f81418a35aa5","stage":"Failed","message":"10 ABORTED: failed to collect enough transaction endorsements, see attached details for more info Details: [{\"address\":\"peer0.org2.example.com:9051\",\"message\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: error while dialing: dial tcp 192.168.0.5:9051: connect: connection refused\\\"\",\"mspId\":\"Org2MSP\"}]"}

{"component":"CLIENT","timestamp":"2021-11-03T13:31:19.580Z","txnId":"ea3f10e176e35a6d8e8ec2be433051bd48f3ba467e0349132171b1649b66330f","stage":"Failed","message":"10 ABORTED: failed to collect enough transaction endorsements, see attached details for more info Details: [{\"address\":\"peer0.org2.example.com:9051\",\"message\":\"rpc error: code = Canceled desc = grpc: the client connection is closing\",\"mspId\":\"Org2MSP\"},{\"address\":\"peer0.org3.example.com:11051\",\"message\":\"rpc error: code = Unavailable desc = transport is closing\",\"mspId\":\"Org3MSP\"}]"}

{"component":"CLIENT","timestamp":"2021-11-03T13:32:35.752Z","txnId":"687cf65618a727ff0d82f2779b6c444d66e1cad002df991486484fe73cf636a1","stage":"Failed","message":"10 ABORTED: failed to collect enough transaction endorsements, see attached details for more info Details: [{\"address\":\"peer1.org3.example.com:11151\",\"message\":\"error 500, error in simulation: failed to execute transaction 687cf65618a727ff0d82f2779b6c444d66e1cad002df991486484fe73cf636a1: error sending: chaincode stream terminated\",\"mspId\":\"Org3MSP\"},{\"address\":\"peer0.org3.example.com:11051\",\"message\":\"error 500, error in simulation: failed to execute transaction 687cf65618a727ff0d82f2779b6c444d66e1cad002df991486484fe73cf636a1: error sending: chaincode stream terminated\",\"mspId\":\"Org3MSP\"}]"}
{"component":"CLIENT","timestamp":"2021-11-03T12:00:57.677Z","txnId":"bad55ec98c0c2c19617fb3a7c0780f8d963430038b75e6ce5ffe4b304ae83eb0","stage":"Failed","message":"14 UNAVAILABLE: no combination of peers can be derived which satisfy the endorsement policy: no peer combination can satisfy the endorsement policy Details: []"}

The messages are typical fabric terminology but don't provide enough detail and you have to resort to logs, have raised an issue for discussion about if there is a way to improve the information for quicker problem determination

davidkel commented 2 years ago

MajorityEP Scenarios

StopRestartOrg2
StopRestartOrg3
StopAPeerInAllOrgs
StopRestartOrg12Peer
StopRestartOrg13Peer
StopRestartOrg23Peer
StopRestartOrg1Peer
StopRestartOrg2Peer
StopRestartOrg3Peer

All EP Scenarios

StopAPeerInAllOrgs
StopRestartOrg12Peer
StopRestartOrg13Peer
StopRestartOrg23Peer
StopRestartOrg1Peer
StopRestartOrg2Peer
StopRestartOrg3Peer