Client TLS handshake failed after 142.128212ms with error: x509: certificate signed by unknown authority

[subhasisbanik]

I am facing Client TLS handshake failure while I am trying to connect from fabric-gateway to HLF network setup.

NodeSDK used: Fabric-gatway version: 1.2.2

HLF Peer version: hyperledger/fabric-peer:2.4.2

Error from the peer logs:

Error from Gateway:

Code invoking the submit transaction:

[subhasisbanik]

@bestbeforetoday Please help if you have any ideas here... Thanks in advance.

[bestbeforetoday]

This issue doesn't seem to be directly related to the Fabric Gateway client API. Instead, it relates to the way the gRPC client connection has been created. Your application code passes the gRPC client connection as an option to the connect() function, used to obtain a Gateway instance. The gRPC runtime can be lazy about establishment of connections so you can see errors only at the point of use, when you actually try to send a request using the connection.

If the CA that signed the peer TLS certificate is not one in your client's trust store, you may need to explicitly specify the CA certificate when creating the gRPC client connection. This is described in the Fabric documentation, and is demonstrated in the fabric samples.

[subhasisbanik]

Hi @bestbeforetoday , Thanks for the response. I went through the documentation provided by you. I observed that the grpc.ssl_target_name_override in my code refers to the same hostname as the one in /etc/hosts.

Can that cause a problem? Or do we need to pass on the public IP in that case ?

Is there a way I can verify if the CA had signed the certificate with public IP or the hostname as in /etc/hosts?

Also the previous version(fabric-network) required tlsCACerts of the certificate authority. Is that required still? If yes, is there any example?

[bestbeforetoday]

As mentioned above, if the CA certificate is not in your trust store, you should explicitly specify the CA when creating the gRPC client connection. In the sample code, this is named tlsCredentials:

const tlsRootCert = await fs.readFile(tlsCertPath);
const tlsCredentials = grpc.credentials.createSsl(tlsRootCert);
return new grpc.Client(peerEndpoint, tlsCredentials);

The grpc.ssl_target_name_override property only needs to be set if the network name/address specified in the peer's TLS certificate does not match the one used by the client to connect to the peer. In a real deployment this should not be the case and you should not need to specify this property. It will be required if the peer is running in a Docker network on your local machine, the peer's TLS certificate specifies its address within the Docker network, but your client is connecting to the peer at a specific localhost port. This is described in the linked documentation:

For a TLS connection to be successfully established, the endpoint address used by the client must match the address in the gateway’s TLS certificate. Since the client accesses the gateway’s Docker container at a localhost address, a gRPC option is specified to force this endpoint address to be interpreted as the gateway’s configured hostname.

The gRPC API documentation might be helpful to see the options available.

[subhasisbanik]

Hi @bestbeforetoday , Thanks for the guidance. I am already setting : tlsRootCert tlsCredentials

as you have mentioned in the example from the start of this issue. Could there be anything else that I am missing?

[subhasisbanik]

Hi @andrew-coleman , I am facing a similar error as : https://github.com/hyperledger/fabric/issues/3224

Could you please help me here?

[andrew-coleman]

When you say you are facing a similar error, are you saying that intermediate certificates are not working for you? Without specific details of your network, and what you have tried, we can only point to general guidance, which is what Mark has already done above.

[bestbeforetoday]

I am facing a similar error as : hyperledger/fabric#3224

Note that, if you are seeing issues with intermediate CA certificates as described in hyperledger/fabric#3224, that was fixed in Fabric peer v2.4.3. In your first comment you state that you are using Fabric peer v2.4.2. Consider upgrading to the latest v2.4 release, or ideally the latest v2.5 release of Fabric, which is the current supported release.

Note also that for full functionality, the current release of the Fabric Gateway client API requires a Fabric peer at v2.4.4 or later. See the Compatibility section of the documentation homepage.

[subhasisbanik]

Thanks @bestbeforetoday and @andrew-coleman I upgraded the peer version to v2.4.4 and it worked seamlessly. I will check the query and the event subscription and close this ticket asap.

[subhasisbanik]

Hi @bestbeforetoday , Peer upgrade was done from v2.4.2 to v2.4.4 via the docker compose change and it was simply restarted. This caused the chaincode containers to also go down and come up when a transaction was invoked. What I observed after couple of transactions is that the chaincode container and the peer container getting down(both from fabric-gateway and cli) and below error in the peer log was observed:

`2023-05-11 15:17:49.055 UTC 0687 ERRO [endorser] simulateProposal -> failed to invoke chaincode samplecontract, error: container exited with 2

github.com/hyperledger/fabric/core/chaincode.(*RuntimeLauncher).Launch.func1

/go/src/github.com/hyperledger/fabric/core/chaincode/runtime_launcher.go:118

runtime.goexit

/usr/local/go/src/runtime/asm_amd64.s:1571

chaincode registration failed

could not launch chaincode samplecontract:a63edebde5704530f347332aea01c342e5b9ec1468ae120c2aba264e8333f470

github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Launch

/go/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:87

github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke

/go/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:203

github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute

/go/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:161

github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute

/go/src/github.com/hyperledger/fabric/core/endorser/support.go:126

github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:120

github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:187

github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:409

github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:350

github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal

/go/src/github.com/hyperledger/fabric/internal/pkg/gateway/gateway.go:40

github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).planFromFirstEndorser.func1

/go/src/github.com/hyperledger/fabric/internal/pkg/gateway/api.go:308

runtime.goexit

/usr/local/go/src/runtime/asm_amd64.s:1571

failed to execute transaction c9816e52fb4a832ab459ef04c69d49826c8fcb496a2d3f4c26a571cd0987ff96

github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult

/go/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:167

github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute

/go/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:162

github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute

/go/src/github.com/hyperledger/fabric/core/endorser/support.go:126

github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:120

github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:187

github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:409

github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal

/go/src/github.com/hyperledger/fabric/core/endorser/endorser.go:350

github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal

/go/src/github.com/hyperledger/fabric/internal/pkg/gateway/gateway.go:40

github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).planFromFirstEndorser.func1

/go/src/github.com/hyperledger/fabric/internal/pkg/gateway/api.go:308

runtime.goexit

/usr/local/go/src/runtime/asm_amd64.s:1571 channel=samplechannel txID=c9816e52

2023-05-11 15:17:49.055 UTC 0688 WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=samplechannel chaincode=samplecontract error="error in simulation: failed to execute transaction c9816e52fb4a832ab459ef04c69d49826c8fcb496a2d3f4c26a571cd0987ff96: could not launch chaincode samplecontract:a63edebde5704530f347332aea01c342e5b9ec1468ae120c2aba264e8333f470: chaincode registration failed: container exited with 2"

2023-05-11 15:17:49.055 UTC 0689 WARN [gateway] func1 -> Endorse call to endorser failed channel=samplechannel chaincode=samplecontract txID=c9816e52fb4a832ab459ef04c69d49826c8fcb496a2d3f4c26a571cd0987ff96 endorserAddress=peer0.org1.com:7051 endorserMspid=org1MSP error="error in simulation: failed to execute transaction c9816e52fb4a832ab459ef04c69d49826c8fcb496a2d3f4c26a571cd0987ff96: could not launch chaincode samplecontract:a63edebde5704530f347332aea01c342e5b9ec1468ae120c2aba264e8333f470: chaincode registration failed: container exited with 2"

2023-05-11 15:17:49.690 UTC 068a ERRO [chaincode] notifyRegistry -> failed to start samplecontract:a63edebde5704530f347332aea01c342e5b9ec1468ae120c2aba264e8333f470 -- peer will not accept external chaincode connection samplecontract:a63edebde5704530f347332aea01c342e5b9ec1468ae120c2aba264e8333f470 (except in dev mode)

2023-05-11 15:18:03.810 UTC 068b WARN [gateway] planFromFirstEndorser -> Endorse call timed out while collecting first endorsement channel=samplechannel chaincode=samplecontract txID=1b53c9ec06fb3b2a64f223de6207a4659ac34ebe3f77b1a086861e73af59818e

2023-05-11 15:18:03.810 UTC 068c INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Endorse grpc.request_deadline=2023-05-11T15:18:03.809Z grpc.peer_address=10.27.4.6:53114 error="rpc error: code = DeadlineExceeded desc = endorsement timeout expired while collecting first endorsement" grpc.code=DeadlineExceeded grpc.call_duration=15.000958989s

2023-05-11 15:18:09.874 UTC 068d INFO [endorser] callChaincode -> finished chaincode: qscc duration: 0ms channel= txID=d0287fca

2023-05-11 15:18:09.874 UTC 068e INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.27.4.6:46984 grpc.code=OK grpc.call_duration=1.030307ms

`

Another observation is that when I downgraded the peer, the initial x509 was back but was working from cli.

Could you please let me know:

1. If the peer upgrade has to be followed according to: https://hlf.readthedocs.io/en/latest/upgrading_your_components.html#upgrade-the-peers
2. Since the above was only version change in docker compose and not according to the process in #1, did this issue occur?

Please help!

[subhasisbanik]

Hi @bestbeforetoday and @andrew-coleman , The upgrade was solved following the documentation from https://hlf.readthedocs.io/en/latest/upgrading_your_components.html#upgrade-the-peers

Hence closing this ticket.

[bestbeforetoday]

Good to hear that you got it working!