Orderer node join -> connect: operation timed out

[jaljaivikbazaar]

Running my infrastructure on AWS EKS cluster using istio

MacBook-Pro network % kubectl get fabricorderernodes.hlf.kungfusoftware.es  -A
NAMESPACE    NAME        STATE     AGE
fabricdemo   ord-node1   RUNNING   8m4s

The moment I try to join, right after Channel Creation, its failing.

MacBook-Pro network % kubectl hlf channel generate --output=mychannel.block --name=mychannel --organizations Org1MSP --organizations Org2MSP --ordererOrganizations OrdererMSP
DEBU[0000] orderers: [0x140003da000]
DEBU[0000] orderer: OrdererMSP
INFO[0003] Peer organizations=[{Org1MSP 0x1400017a000 0x14000378c00} {Org1MSP 0x1400017c100 0x1400017a580} {Org2MSP 0x1400017d700 0x1400017cc00} {Org2MSP 0x1400051a000 0x14000378680}]
INFO[0003] Orderer organizations=[{OrdererMSP 0x14000377600 0x14000377b80 [hlf-dev-ord-node1.XXXXXXXX.com:443]}]

MacBook-Pro network % kubectl hlf ordnode join \
            --block=mychannel.block \
            --name=ord-node1 \
            --namespace=${NAMESPACE} \
            --identity=admin-tls-ordservice.yaml
INFO[0000] name=ord-node1 namespace=fabricdemo
Error: Post "https://192.168.151.72:31151/participation/v1/channels": dial tcp 192.168.151.72:31151: connect: operation timed out

[dviejokfs]

For the admin hosts of the orderer, you need to use Istio if you are deploying on a Cloud provider since you don't have direct access to the nodes.

Check this guide https://labs.hyperledger.org/hlf-operator/docs/operator-guide/istio

[EphraimNetWorks]

Hi @dviejokfs I ran into the same issue with Istio setup as specified in the operator guide. I have been able to create a working network in minikube and currently trying to deploy to Azure AKS with Istio configuration. Everything else works fine(Network components creation, User registration and enrollment and Channel creation). I get a similar error to @jaljaivikbazaar logs when I try to join an orderer to the channel it fails with connection timed out

Creating channel channel1
DEBU[0001] orderers: [0xc0004c1100 0xc0005c4a00 0xc0004c0300] 
DEBU[0001] orderer: OrdererMSP                          
DEBU[0002] orderer: OrdererMSP                          
DEBU[0002] orderer: OrdererMSP                          
INFO[0006] Peer organizations=[{Org1MSP 0xc0000f2100 0xc0000f0b00} {Org1MSP 0xc0000f2c00 0xc0000f0580}] 
INFO[0006] Orderer organizations=[{OrdererMSP 0xc0002f7b80 0xc0002f8680 [node1.orderer.<<domain>>:443 node2.orderer.<<domain>>:443 node3.orderer.<<domain>>:443]}] 
 [fabsdk/fab] 2022/09/28 15:34:26 UTC - n/a -> INFO TLS Enabled
 [fabsdk/fab] 2022/09/28 15:34:26 UTC - n/a -> INFO generating key: &{A:ecdsa S:256}
 [fabsdk/fab] 2022/09/28 15:34:26 UTC - logbridge.(*cLogger).Info -> INFO encoded CSR
Joining orderer-node1 to channel1
INFO[0000] name=orderer-node1 namespace=fabric  
Error: Post "https://10.224.0.4:30547/participation/v1/channels": dial tcp 10.224.0.4:30547: connect: connection timed out

Below is the deployment yaml

apiVersion: hlf.kungfusoftware.es/v1alpha1
kind: FabricOrdererNode
metadata:
  creationTimestamp: null
  name: orderer-node1
  namespace: fabric
spec:
  adminIstio:
    ingressGateway: ingressgateway
    port: 443
  affinity: null
  bootstrapMethod: none
  channelParticipationEnabled: true
  env: null
  genesis: ""
  grpcProxy: null
  hostAliases: []
  image: hyperledger/fabric-orderer
  imagePullSecrets: null
  istio:
    hosts:
    - node1.orderer.<<domain>>
    ingressGateway: ingressgateway
    port: 443
  mspID: OrdererMSP
  pullPolicy: Always
  replicas: 1
  resources: {}
  secret:
    enrollment:
      component:
        cahost: ca.orderer.<<domain>>
        caname: ca
        caport: 443
        catls:
          cacert: <<cert>>
        enrollid: orderer
        enrollsecret: ordererpw
      tls:
        cahost: ca.orderer.<<domain>>
        caname: tlsca
        caport: 443
        catls:
          cacert: <<cert>>
        csr:
          cn: ""
          hosts:
          - 127.0.0.1
          - localhost
          - 10.224.0.4
          - orderer-node1
          - orderer-node1.fabric
          - node1.orderer.<<domain>>
        enrollid: orderer
        enrollsecret: ordererpw
  service:
    type: NodePort
  serviceMonitor: null
  storage:
    accessMode: ReadWriteOnce
    size: 2Gi
    storageClass: default
  tag: 2.4.3
  tolerations: null
  updateCertificateTime: null

[EphraimNetWorks]

@jaljaivikbazaar did you figure out what the problem was?

[EphraimNetWorks]

After days of debugging, found that the issue is resolved by setting --admin-hosts when creating the orderer. Also had to upgrade the operator version from 1.7.0 to 1.8.0-beta12. I'm sure issue will be fully resolved when version 1.8.0 is released fully and documentation is updated.

#upgrading hlf to devel release
helm delete hlf-operator
helm repo add kfs https://kfsoftware.github.io/hlf-helm-charts --force-update 
helm install hlf-operator --devel --version="1.8.0-beta12" kfs/hlf-operator

#creating orderer
#remember to add A records for the orderer admin hosts also pointing to the Istio LoadBalancer external IP
kubectl hlf ordnode create --image=${ORDERER_IMAGE} \
        --version=${ORDERER_VERSION} \
        --storage-class=${STORAGE_CLASS} \
        --enroll-id=${ORDERER_ENROLL_ID} \
        --mspid=${ORDERER_CA_MSP_ID} \
        --enroll-pw=${ORDERER_ENROLL_PW} \
        --capacity=${ORDERER_CAPACITY} \
        --name=${ORDERER_NAME} \
        --admin-hosts=$ORDERER_ADMIN_DOMAIN \
        --ca-name=${ORDERER_NAMESPACE_CA_NAME} \
        --ca-host=${ORDERER_CA_DOMAIN} \
        --ca-port=${ISTIO_GW_PORT} \
        --hosts="$ORDERER_DOMAIN" \
        --istio-ingressgateway=$ISTIO_INGRESSGATEWAY \
        --istio-port=$ISTIO_GW_PORT \
        --namespace=${HLF_NAMESPACE} \
        --output > $ORDERER_NAME-deployment.yaml

    yq -i "
        .spec.secret.enrollment.tls.csr.hosts[6] |= \"$ORDERER_ADMIN_DOMAIN\"
    " $ORDERER_NAME-deployment.yaml
    kubectl apply -f $ORDERER_NAME-deployment.yaml

[dviejokfs]

Closing the issue, if this happens again please open another one.

[pshgpt]

@EphraimNetWorks I'm also facing this same issue. Could you please tell me what is the value for $ORDERER_ADMIN_DOMAIN