hyperledger-bevel / bevel

An automation framework for rapidly and consistently deploying production-ready DLT platforms
https://hyperledger-bevel.readthedocs.io/en/latest/
Apache License 2.0
346 stars 719 forks source link

When I try to create an environment in minikube, the Hashcorp vault authentication does not pass. #2630

Open atomic-kanta-sasaki opened 2 weeks ago

atomic-kanta-sasaki commented 2 weeks ago

Describe the bug When I try to create an environment in minikube, the Hashcorp vault authentication does not pass.

Hi. https://hyperledger-bevel.readthedocs.io/en/latest/tutorials/bevel-minikube-setup/ I am attempting to deploy Hyperledger Fabric on minikube with this documentation. However, I cannot access the Hashcorp Vault and am unable to complete the task.

When I check the logs as follows, I get this kubectl logs -n manufacturer-net fabric-ca-server-ca-0 -c ca-certs-init

curl: (7) Failed to connect to 172.17.220.191 port 8200 after 3129 ms: Couldn't connect to server
Vault token API call response: {"errors":["permission denied"]}
Error: Failed to obtain Vault token.
Error Details: permission denied

Is there anything else I need to do regarding Hashcorp Vault other than the following commands?

export VAULT_ADDR='http://<Your Vault local IP address>:8200' #e.g. http://192.168.0.1:8200
export VAULT_TOKEN="<Your Vault root token>"

# enable Secrets v2
vault secrets enable -version=2 -path=secretsv2 kv   

Environment (please complete the following information):

michaelkhalsa commented 1 week ago

Hi @atomic-kanta-saski

To verify: can you open the vault from a browser, and check that the secretsv2 engine was created?

When you create the vault, use a local config.hcl file, where you tell it to add a ui
example (for a localhost instance):

ui = true
storage "file" {
  path    = "./bevel/data"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

For your future info:
Once the playbook can access the vault, it wil create an ACL policy  in the vault if not already present.
 example: bevel-vault-mgmt-ca-\-net-policy
If you recreated the cluster and used a different network.env.type name in the network.yaml, than you may need to manually update the policy to point to the new path, as the playbook only checks for its existance, not the value of the policy.

Example of policy if using network.env.type = dev
If i later changed it to dev2 to recreate everything, be sure to manually update this to ../data/dev2\/*
for each of yhour orgs (each org has its own policy)

 path "secretsv2/data/devmainorg/*" {      capabilities = ["list", "read", "create", "update"]    }    path "secretsv2/data/devmainorg/credentials/*" {      capabilities = ["list", "read", "create", "update"]    }

ka-sasaki-sti commented 1 week ago

@michaelkhalsa Thank you very much. This issue has been resolved. However, we are currently facing the following problem When I run the following command kubectl describe pod fabric-orderernode-orderer1-0 -n supplychain-net I get an error like this

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  11m                   default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         11m                   default-scheduler  Successfully assigned supplychain-net/fabric-orderernode-orderer1-0 to minikube
  Warning  FailedMount       9m32s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[orderer1-genesis-volume], unattached volumes=[orderer1-genesis-volume kube-api-access-rc4ds certificates scripts-volume datadir]: timed out waiting for the condition
  Warning  FailedMount       7m17s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[orderer1-genesis-volume], unattached volumes=[scripts-volume datadir orderer1-genesis-volume kube-api-access-rc4ds certificates]: timed out waiting for the condition
  Warning  FailedMount       2m46s (x2 over 5m3s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[orderer1-genesis-volume], unattached volumes=[kube-api-access-rc4ds certificates scripts-volume datadir orderer1-genesis-volume]: timed out waiting for the condition
  Warning  FailedMount       74s (x13 over 11m)    kubelet            MountVolume.SetUp failed for volume "orderer1-genesis-volume" : configmap "syschannel-genesis" not found
  Warning  FailedMount       29s                   kubelet            Unable to attach or mount volumes: unmounted volumes=[orderer1-genesis-volume], unattached volumes=[datadir orderer1-genesis-volume kube-api-access-rc4ds certificates scripts-volume]: timed out waiting for the condition

It seems to be caused by the configmap not being created. Why is the configmap not created?

./run.sh logs

fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "vault kv get -field=rootca_key secretsv2/localsupplychain/ca\n", "delta": "0:00:00.053154", "end": "2024-11-14 04:23:02.918095", "msg": "non-zero return code", "rc": 2, "start": "2024-11-14 04:23:02.864941", "stderr": "Get \"http://host.minikube.internal:8200/v1/sys/internal/ui/mounts/secretsv2/localsupplychain/ca\": dial tcp: lookup host.minikube.internal on 8.8.8.8:53: no such host", "stderr_lines": ["Get \"http://host.minikube.internal:8200/v1/sys/internal/ui/mounts/secretsv2/localsupplychain/ca\": dial tcp: lookup host.minikube.internal on 8.8.8.8:53: no such host"], "stdout": "", "stdout_lines": []}
michaelkhalsa commented 1 week ago

Hi @atomic-kanta-saski

Are you referring to configmap: orderer1-config in ns supplychain-net?

While others may be able to answer this better, it looks like some kind of permissions issue with minikube, wsl2, and the windows os. (assuming using ubuntu on wsl2). I vaguely recall having mount issues with minikube and WSL2 based on permissions when trying that route.

Maybe try running the playbook with sudo (check that your config file is same with sudo first), and if that does not help, then browse for mount issues, such as  'mountvolume.setup minikube wsl2'.

did you assign enough ram and cpus to the kube? 
enough storage on your computer?
enough virtual resources for the kube from your host os?

At what task did it fail while running the playbook?

--
If using flux, did it write the values file for the orderer in your repository
example: .../releases/dev/supplychain/orderer/orderer1.yaml

Also if you run kubectl logs \ -n \
It should show you the values used for the helm chart (assuming it did not get that far)
and if pending initializing with a system channel?

Michael

atomic-kanta-sasaki commented 3 days ago

@michaelkhalsa thanks! I have encountered a new problem.

The error is as follows.

TASK [/home/bevel/platforms/hyperledger-fabric/configuration/../../shared/configuration/roles/create/job_component : Helm install genesis] ***********************************************************************************************
task path: /home/bevel/platforms/shared/configuration/roles/create/job_component/tasks/main.yaml:39
fatal: [localhost]: FAILED! => {"changed": false, "command": "/root/bin/helm repo update", "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: no repositories found. You must add one before updating\n", "stderr": "Error: no repositories found. You must add one before updating\n", "stderr_lines": ["Error: no repositories found. You must add one before updating"], "stdout": "", "stdout_lines": []}

I am error that not a single helm has been added to the repository, is this a recognized error? I changed my development environment from WSL to MacOS and the previous error has been resolved.