hyperledger-bevel / bevel-operator-fabric

Hyperledger Fabric Kubernetes operator - Hyperledger Fabric operator for Kubernetes (v2.3, v2.4 and v2.5, soon 3.0)
https://hyperledger-bevel.github.io/bevel-operator-fabric/
Apache License 2.0
280 stars 93 forks source link

Chaincode uses cluster IP of peers and unable to connect if cluster restart cause of Pod cluster IP change. #40

Closed aim8526 closed 2 years ago

aim8526 commented 2 years ago

Hi Team,

Architecture, Single Organization = 1 Kubernetes Cluster (Kind on 1 Ec2 Instances) 2 Organization = (1 Ca, 2 peer (leveldb as statedb) , 1 orderer) each organization.

I am trying to deploy a fabric network on the Kubernetes cluster.

By following your meet-up tutorial I am able to set up 2 organizations in a single Kubernetes cluster based on KIND.

It works well chaincode install and initiate properly. thanks for your proper documentation.

But When I restart chain-code pods not running, after describing pods, I realized the cluster IP of the peer is changed and chaincode is unable to connect peer.

Q1 how can I use the service name of peer and order to perform communication locally?

Architecture, Single Organization = 1 Kubernetes Cluster (Microk8s on 3 Ec2 Instances(1 master, 2 worker)) 1 Organization = 1 Ca, 2 peer, 1 orderer.

Q2 I want to create an organization on a Single microK8s cluster & expose PEER, CA, Order, and Couchdb on external IP, So I am able to connect SDK if deployed on another cluster, add another organization available on another Cluster or Network.

I tried adding the domain in ca.yaml host keyword but it does not work and the CA client is unable to communicate with the domain IP. please help me to configure Istio based communication on my fabric network.

Mentioned Host like this in ca. kubectl hlf ca create --name=org1-ca --capacity=2Gi --namespace default --storage-class=microk8s-hostpath \ --hosts ec2-54-69-44-21.us-west-2.compute.amazonaws.com \ --enroll-id=enroll --enroll-pw=enrollpw --output > ca.yaml

But Not able to enroll users.

Mentioned Host like this in Perr. kubectl hlf peer create --name=org1-peer0 --ca-name=org1-ca.default \ --enroll-id=peer --enroll-pw=peerpw \ --capacity=5Gi --storage-class=microk8s-hostpath \ --mspid=Org1MSP --statedb couchdb \ --leader false --hosts ec2-54-69-44-21.us-west-2.compute.amazonaws.com \ --output > peer1.yaml

dviejokfs commented 2 years ago

Hi @aim8526

For the enrolling of the users

When you set up the network using KinD, you have access to the nodes of Kubernetes and the pods, but when you are in AWS or any other cloud provider this is usually not the case.

To work around that, the best solution is to have Istio as a load balancer, why? Because with only one IP you can handle the requests for all the peers, orderers and CAs that you create in that cluster.

You can see this guide: Istio HLF-operator setup

For the peer IP

Not sure what version of the operator you are using, with the latest release, 1.4.0, the cluster IP was replaced by the service name, so this won't happen, see the following line in the variables of the peer: https://github.com/kfsoftware/externalbuilder/blob/main/cmd/launcher/run.go#L131-L139

External builder issue upon restarting the chaincode

There's another issue that is related to the certificates since the chaincode certificates that the peer generate are short-lived. The way to work around this is for the external builder to kill the pod of the chaincode if it's already running before creating the pod for the chaincode, see the external builder code:

https://github.com/kfsoftware/externalbuilder/blob/main/cmd/launcher/run.go#L131-L139

Not sure if I'm missing anything else, please, let us know how it progresses.

dviejokfs commented 2 years ago

Please @aim8526 reopen if necessary.