search

hyperledger-bevel / bevel-operator-fabric

Hyperledger Fabric Kubernetes operator - Hyperledger Fabric operator for Kubernetes (v2.3, v2.4 and v2.5, soon 3.0)
https://hyperledger-bevel.github.io/bevel-operator-fabric/
Apache License 2.0
280 stars 93 forks source link

Operator doesn't create peer (using nodeport) #155

Closed Nova38 closed 1 year ago

Nova38 commented 1 year ago

What happened?

I had some difficulty getting the instructions for the ISTIO setup; namely getting it to work when accessing Istio gateway via nodeport as I don't have a load balancer. (I was having some issues trying to get the dns to work with the mutli-host cluster). I tore down the cluster and brought it back up to see if i could get it to work with node ports instead. I was able to get it to successfully create the CA's and register the users. However, when i tried to create a peer it ended up the operator ends up in a loop and never creates the peer.

(seems similar to issue #23 but the solution was not included in the issue)

What did you expect to happen?

For it to create the specified peer node

How can we reproduce it (as minimally and precisely as possible)?

I have a small cluster of 4 ubuntu 22.04 boxes running Kubernetes though canonical microk8s Kubernetes tool. I was able to replicate the issue after reinstalling the hlf operator from scratch and then running the following commands:

kubectl hlf ca create --name=org1-ca     --storage-class=standard --capacity=1Gi      --enroll-id=enroll --enroll-pw=enrollpw

kubectl hlf ca register --name=org1-ca --type=peer --name=org1-ca --type=peer \
    --user=org1-peer1 --secret=peerpw  \
    --enroll-id enroll --enroll-secret=enrollpw --mspid Org1MSP \
    --namespace=fabric

kubectl hlf peer create --name=org1peer0 --ca-name=org1-ca.fabric \
        --statedb=couchdb --image=hyperledger/fabric-peer  --version=2.4.6  --storage-class=standard \
        --enroll-id=org1-peer0  --enroll-pw=peerpw \
        --capacity=5Gi   --mspid=Org1MSP \
        --namespace=fabric

Anything else we need to know?

Today's log after removing hlf operator an reinstall it.

1.6783453997656748e+09  INFO    controllers.FabricPeer  Service org1peer0 created   {"hlf": "default/org1peer0"}
 [fabsdk/fab] 2023/03/09 07:03:19 UTC - n/a -> INFO TLS Enabled
 [fabsdk/fab] 2023/03/09 07:03:19 UTC - n/a -> INFO generating key: &{A:ecdsa S:256}

Yesterdays log when i tried to make 4 peers and ran into the same issue.

2023-03-08T13:37:56.640567970-06:00  [fabsdk/fab] 2023/03/08 19:37:56 UTC - n/a -> INFO TLS Enabled
2023-03-08T13:37:56.640620768-06:00  [fabsdk/fab] 2023/03/08 19:37:56 UTC - n/a -> INFO generating key: &{A:ecdsa S:256}
2023-03-08T13:37:56.641420569-06:00  [fabsdk/fab] 2023/03/08 19:37:56 UTC - logbridge.(*cLogger).Info -> INFO encoded CSR
2023-03-08T13:37:56.824472014-06:00 1.6783042768241022e+09  ERROR   controllers.FabricPeer  k8sAPIClientError failed to update the application status   {"error": "Operation cannot be fulfilled on fabricpeers.hlf.kungfusoftware.es \"peer1-org2\": the object has been modified; please apply your changes to the latest version and try again"}
2023-03-08T13:37:56.824559839-06:00 github.com/kfsoftware/hlf-operator/controllers/peer.(*FabricPeerReconciler).Reconcile
2023-03-08T13:37:56.824588681-06:00     github.com/kfsoftware/hlf-operator/controllers/peer/peer_controller.go:485
2023-03-08T13:37:56.824607715-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2023-03-08T13:37:56.824625692-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
2023-03-08T13:37:56.824642145-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2023-03-08T13:37:56.824660829-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
2023-03-08T13:37:56.824673122-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2023-03-08T13:37:56.824684845-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
2023-03-08T13:37:56.824696271-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2023-03-08T13:37:56.824707362-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-03-08T13:37:56.824759298-06:00 1.6783042768243346e+09  ERROR   Reconciler error    {"controller": "fabricpeer", "controllerGroup": "hlf.kungfusoftware.es", "controllerKind": "FabricPeer", "fabricPeer": {"name":"peer1-org2","namespace":"fabric"}, "namespace": "fabric", "name": "peer1-org2", "reconcileID": "925b683e-dc8e-4e94-9c53-75b6660b4e25", "error": "Operation cannot be fulfilled on fabricpeers.hlf.kungfusoftware.es \"peer1-org2\": the object has been modified; please apply your changes to the latest version and try again"}
2023-03-08T13:37:56.824785840-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2023-03-08T13:37:56.824805876-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
2023-03-08T13:37:56.824824685-06:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2023-03-08T13:37:56.824842687-06:00     sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-03-08T13:37:57.668213160-06:00 1.6783042776679947e+09  INFO    controllers.FabricPeer  Successfully finalized peer {"hlf": "fabric/peer1-org2"}
2023-03-08T13:37:57.894184062-06:00 1.6783042778939083e+09  INFO    controllers.FabricPeer  Service peer1-org2 couldn't be found    {"hlf": "fabric/peer1-org2"}
2023-03-08T13:37:57.894270674-06:00 1.6783042778939703e+09  INFO    controllers.FabricPeer  Successfully finalized peer {"hlf": "fabric/peer1-org2"}
2023-03-08T13:37:57.972546572-06:00 1.6783042779722624e+09  ERROR   controllers.FabricPeer  k8sAPIClientError failed to update the application status   {"error": "Operation cannot be fulfilled on fabricpeers.hlf.kungfusoftware.es \"peer1-org2\": StorageError: invalid object, Code: 4, Key: /registry/hlf.kungfusoftware.es/fabricpeers/fabric/peer1-org2, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f032cadb-d06d-47a2-ab1f-7b26c42bc042, UID in object meta: "}

Kubernetes version

The nodes are using microk8s from canonical. ```console # kubectl get nodes -o wide $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME redwood.ittc.ku.edu Ready 3d22h v1.26.1 129.237.120.240 Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.8 pengo.ittc.ku.edu Ready 3d22h v1.26.1 129.237.120.78 Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.8 fzero.ittc.ku.edu Ready 3d22h v1.26.1 129.237.123.11 Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.8 rafflesia.ittc.ku.edu Ready 3d22h v1.26.1 129.237.121.77 Ubuntu 22.04.2 LTS 5.15.0-67-generic containerd://1.6.8 ``` Also ```console microk8s status microk8s is running high-availability: yes datastore master nodes: 129.237.120.240:19001 129.237.121.77:19001 129.237.123.11:19001 datastore standby nodes: 129.237.120.78:19001 addons: enabled: community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes host-access # (core) Allow Pods connecting to Host services smoothly hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access metrics-server # (core) K8s Metrics Server for API access to service metrics observability # (core) A lightweight observability stack for logs, traces and metrics storage # (core) Alias to hostpath-storage add-on, deprecate ```
Nova38 commented 1 year ago

I also am occationally getting this error when I try to do the first two commands. Note that the host i am running the commands on has the ip address of 129.237.120.240 and the host it deployed on had 129.237.120.78. Interestingly it then worked 5 minutes latter. I had waited for the running condition to work: kubectl wait --timeout=180s --for=condition=Running fabriccas.hlf.kungfusoftware.es --all. I included the hlf inspects when it didn't work and then after it started working.

HLF ca register failed ``` console $ kubectl hlf ca register --name=org1-ca --type=peer --name=org1-ca --type=peer --user=org1-peer1 --secret=peerpw --enroll-id enroll --enroll-secret=enrollpw --mspid Org1MSP [fabsdk/fab] 2023/03/09 07:34:33 UTC - n/a -> INFO TLS Enabled [fabsdk/fab] 2023/03/09 07:34:33 UTC - n/a -> INFO generating key: &{A:ecdsa S:256} [fabsdk/fab] 2023/03/09 07:34:33 UTC - logbridge.(*cLogger).Info -> INFO encoded CSR Error: enroll failed: enroll failed: POST failure of request: POST https://129.237.120.78:30986/enroll {"hosts":null,"certificate_request":"-----BEGIN CERTIFICATE REQUEST-----\nMIH9MIGkAgEAMBExDzANBgNVBAMTBmVucm9sbDBZMBMGByqGSM49AgEGCCqGSM49\nAwEHA0IABGkhR1Wgk7m9T7b9TmhuvT/w7c+A1RnBFuEy3tOGBIfiKOJtGfmBxKyx\ndIs1RsmR2UBbVaEetQ3w7uTdPf1kNY2gMTAvBgkqhkiG9w0BCQ4xIjAgMB4GA1Ud\nEQQXMBWCE3JlZHdvb2QuaXR0Yy5rdS5lZHUwCgYIKoZIzj0EAwIDSAAwRQIhAMen\nCRXJ+R4TEajgamjs/HuMCirP/5mIoaeY+iI2Up9JAiAFK3lo7SBSq5uSkQJDN2AM\no2BIQ5pEoWILVm6zxOzRiw==\n-----END CERTIFICATE REQUEST-----\n","profile":"","crl_override":"","label":"","NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","ReturnPrecert":false,"CAName":""}: Post "https://129.237.120.78:30986/enroll": x509: certificate is valid for 127.0.0.1, 129.237.120.240, not 129.237.120.78 ```
HLF Inspect outputs First output ``` [fabsdk/fab] 2023/03/09 07:34:47 UTC - n/a -> INFO Found 0 organizations name: hlf-network version: 1.0.0 client: organization: "" organizations: {} orderers: [] peers: {} certificateAuthorities: org1-ca.default: url: https://129.237.120.78:30986 registrar: enrollId: enroll enrollSecret: enrollpw caName: ca tlsCACerts: pem: - | -----BEGIN CERTIFICATE----- MIICTzCCAfSgAwIBAgIRAJ0vjxHcPDItgtMUAMPAw2kwCgYIKoZIzj0EAwIwUjET MBEGA1UEBhMKQ2FsaWZvcm5pYTEJMAcGA1UEBxMAMQkwBwYDVQQJEwAxFDASBgNV BAoTC0h5cGVybGVkZ2VyMQ8wDQYDVQQLEwZGYWJyaWMwHhcNMjMwMzA4MDczMzM5 WhcNMzMwMzA5MDczMzM5WjBSMRMwEQYDVQQGEwpDYWxpZm9ybmlhMQkwBwYDVQQH EwAxCTAHBgNVBAkTADEUMBIGA1UEChMLSHlwZXJsZWRnZXIxDzANBgNVBAsTBkZh YnJpYzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABN0fvskO6cs2+5xsON2QMr6W Ww5J7FpeJz2WZXU+oXOthztRm4skRn1DAPualLVpZPgQEtbz9PUjSe1uS6goZRWj gaowgacwDgYDVR0PAQH/BAQDAgGmMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDATAPBgNVHRMBAf8EBTADAQH/MCkGA1UdDgQiBCCQkYUIgRkTfUxntPC4QEFj gr8wbzpMxCYb5/D25URrpzA6BgNVHREEMzAxgglsb2NhbGhvc3SCB29yZzEtY2GC D29yZzEtY2EuZGVmYXVsdIcEfwAAAYcEge148DAKBggqhkjOPQQDAgNJADBGAiEA 8e81qzr9QDUOsE5isMvIhFw186p9VkBBkjhUtK5LONoCIQDYS37RRBZE/hsypXyD McOGigQ7mT8JIGcmHxH2vnp05w== -----END CERTIFICATE----- channels: _default: orderers: [] peers: {} ``` Output after a few minutes that after it changed to let me register the users ``` [fabsdk/fab] 2023/03/09 07:38:28 UTC - n/a -> INFO Found 0 organizations name: hlf-network version: 1.0.0 client: organization: "" organizations: {} orderers: [] peers: {} certificateAuthorities: org1-ca.default: url: https://129.237.120.240:30986 registrar: enrollId: enroll enrollSecret: enrollpw caName: ca tlsCACerts: pem: - | -----BEGIN CERTIFICATE----- MIICTzCCAfSgAwIBAgIRAJ0vjxHcPDItgtMUAMPAw2kwCgYIKoZIzj0EAwIwUjET MBEGA1UEBhMKQ2FsaWZvcm5pYTEJMAcGA1UEBxMAMQkwBwYDVQQJEwAxFDASBgNV BAoTC0h5cGVybGVkZ2VyMQ8wDQYDVQQLEwZGYWJyaWMwHhcNMjMwMzA4MDczMzM5 WhcNMzMwMzA5MDczMzM5WjBSMRMwEQYDVQQGEwpDYWxpZm9ybmlhMQkwBwYDVQQH EwAxCTAHBgNVBAkTADEUMBIGA1UEChMLSHlwZXJsZWRnZXIxDzANBgNVBAsTBkZh YnJpYzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABN0fvskO6cs2+5xsON2QMr6W Ww5J7FpeJz2WZXU+oXOthztRm4skRn1DAPualLVpZPgQEtbz9PUjSe1uS6goZRWj gaowgacwDgYDVR0PAQH/BAQDAgGmMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDATAPBgNVHRMBAf8EBTADAQH/MCkGA1UdDgQiBCCQkYUIgRkTfUxntPC4QEFj gr8wbzpMxCYb5/D25URrpzA6BgNVHREEMzAxgglsb2NhbGhvc3SCB29yZzEtY2GC D29yZzEtY2EuZGVmYXVsdIcEfwAAAYcEge148DAKBggqhkjOPQQDAgNJADBGAiEA 8e81qzr9QDUOsE5isMvIhFw186p9VkBBkjhUtK5LONoCIQDYS37RRBZE/hsypXyD McOGigQ7mT8JIGcmHxH2vnp05w== -----END CERTIFICATE----- channels: _default: orderers: [] peers: {} ```
Nova38 commented 1 year ago

I also tried giving it the host's ip as --ca-host and the port as --ca-port. (i also tried leaving off the ca-name and swapping the ip address for the ca-name but both errored out)

kubectl hlf peer create --name=org1-peer0 \
        --ca-name=org1-ca.default --ca-host=129.237.120.240 --ca-port=30986 \
        --statedb=couchdb --image=hyperledger/fabric-peer  --version=2.4.6  --storage-class=standard \
        --enroll-id=org1-peer0  --enroll-pw=peerpw \
        --capacity=5Gi   --mspid=Org1MSP