search

kubernetes-sigs / ibm-vpc-block-csi-driver

ibm-vpc-block-csi-driver is a CSI plugin for creating and mounting VPC block storage on IBM VPC infrastructure based openshift or kubernetes cluster
Apache License 2.0
4 stars 27 forks source link

The deploy-csi-driver-on-your-cluster section in Readme.md need be updated. #143

Closed liudalibj closed 6 months ago

liudalibj commented 1 year ago

When I follow https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/blob/master/README.md#deploy-csi-driver-on-your-cluster to install ibm-vpc-block-csi-driver to my IKS cluster.

I meet follow error:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
Error: invalid Kustomization: json: cannot unmarshal string into Go struct field Kustomization.patches of type types.Patch
error: no objects passed to apply
root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver#

I tried to fix the issue by replace the patches: to patchesStrategicMerge: in deploy/kubernetes/driver/kubernetes/overlays/stage/kustomization.yaml

The error message is changed to:

bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Error: no matches for Id StatefulSet.v1.apps/ibm-vpc-block-csi-controller.[noNs]; failed to find unique target for patch StatefulSet.v1.apps/ibm-vpc-block-csi-controller.[noNs]
error: no objects passed to apply

I tried to fix the new issue by add namespace namespace: kube-system to controller-server-images.yaml and node-server-images.yaml under deploy/kubernetes/driver/kubernetes/overlays/stage the new error looks like:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Error: no matches for Id StatefulSet.v1.apps/ibm-vpc-block-csi-controller.kube-system; failed to find unique target for patch StatefulSet.v1.apps/ibm-vpc-block-csi-controller.kube-system
error: no objects passed to apply

I tried to fix the new issue by update the StatefulSet to Deployment of deploy/kubernetes/driver/kubernetes/overlays/stage/controller-server-images.yaml The new error looks like:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/ibm-vpc-block-controller-sa created
serviceaccount/ibm-vpc-block-node-sa created
clusterrole.rbac.authorization.k8s.io/vpc-block-driver-registrar-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-driver-snapshotter-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-external-attacher-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-external-resizer-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-driver-registrar-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-driver-snapshotter-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-external-attacher-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-external-resizer-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-provisioner-binding created
configmap/addon-vpc-block-csi-driver-configmap created
configmap/ibm-vpc-block-csi-configmap created
secret/storage-secret-store created
csidriver.storage.k8s.io/vpc.block.csi.ibm.io created
Error from server (BadRequest): error when creating "STDIN": Deployment in version "v1" cannot be handled as a Deployment: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'
Error from server (BadRequest): error when creating "STDIN": DaemonSet in version "v1" cannot be handled as a DaemonSet: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

Try to find the built out yaml file, find that the resources section broken the deploy, there are many {{kube-system.addon-vpc-block-csi-driver-configmap.xxx}} need be replaced?

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# kustomize build deploy/kubernetes/driver/kubernetes/overlays/stage/
...
        resources:
          limits:
            cpu: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}120m{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}'
            memory: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}300Mi{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}'
          requests:
            cpu: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}30m{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}'
            memory: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}75Mi{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}'
...

I install kustomize by command

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

the version is 'v5.0.3'

The go path

GOPATH=/root/go
GunaKKIBM commented 1 year ago

Hi @liudalibj .

Actually, that is mustache templating and it won't work with all the braces and reference with config file, for the time being please replace it with the default cpu and memory values which is already specified there. We will raise a PR to fix it with default values.

liudalibj commented 1 year ago

Thanks the reply @GunaKKIBM , I tried to replace all {{}} with empty string so that use the default values of cpu and memory.

pattern="\{\{[^}]*\}\}"
sed -E -i "s/$pattern//g" "$filepath"

The csi-driver pods are created with the fixed deployment yaml file. But the ibm-vpc-block-csi containers are failed to start.

kube-system    ibm-vpc-block-csi-controller-bdfdf4657-qgdvr       5/6     CrashLoopBackOff   11 (4m32s ago)   36m
kube-system    ibm-vpc-block-csi-node-tfpb6                       2/3     CrashLoopBackOff   11 (4m43s ago)   36m
kube-system    ibm-vpc-block-csi-node-tnnsb                       2/3     CrashLoopBackOff   11 (4m37s ago)   36m

the controller logs:

root@liudali-csi-amd64-node-0:~# kubectl logs -n kube-system    ibm-vpc-block-csi-controller-bdfdf4657-qgdvr -c iks-vpc-block-driver
{"level":"info","timestamp":"2023-06-16T05:57:31.418Z","caller":"cmd/main.go:87","msg":"IBM CSI driver version","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","DriverVersion":"vpcBlockDriver-"}
{"level":"info","timestamp":"2023-06-16T05:57:31.418Z","caller":"cmd/main.go:88","msg":"Controller Mutex Lock enabled","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","LockEnabled":false}
{"level":"info","timestamp":"2023-06-16T05:57:31.419Z","caller":"ibmcloudprovider/volume_provider.go:50","msg":"NewIBMCloudStorageProvider-Reading provider configuration...","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver"}
{"level":"error","timestamp":"2023-06-16T05:57:31.428Z","caller":"config/config.go:172","msg":"Failed to parse config","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","error":"toml: line 9 (last key \"vpc\"): expected a top-level item to end with a newline, comment, or EOF, but got '0' instead"}
{"level":"error","timestamp":"2023-06-16T05:57:31.428Z","caller":"config/config.go:62","msg":"Error parsing config","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","error":"toml: line 9 (last key \"vpc\"): expected a top-level item to end with a newline, comment, or EOF, but got '0' instead"}
{"level":"error","timestamp":"2023-06-16T05:57:31.428Z","caller":"ibmcloudprovider/volume_provider.go:54","msg":"Error loading configuration","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver"}
{"level":"fatal","timestamp":"2023-06-16T05:57:31.428Z","caller":"cmd/main.go:97","msg":"Failed to instantiate IKS-Storage provider","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","error":"toml: line 9 (last key \"vpc\"): expected a top-level item to end with a newline, comment, or EOF, but got '0' instead"}

It seems that the master code failed to deal the file from https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/blob/master/deploy/kubernetes/driver/kubernetes/slclient_Gen2.toml?

Check the latest config format https://github.com/IBM/secret-utils-lib/blob/master/secrets/storage-secret-store/slclient.toml, maybe we should use

...
[VPC]
...

?

liudalibj commented 1 year ago

csi-containers started with slclient_Gen2.toml as follow:

[VPC]
  iam_client_id = "bx"
  iam_client_secret = "bx"
  g2_token_exchange_endpoint_url = "https://iam.cloud.ibm.com"
  g2_riaas_endpoint_url = "https://eu-gb.iaas.cloud.ibm.com"
  g2_resource_group_id = "0013399570e34af29d51788b600fc617"
  g2_api_key = "myapikey"
  provider_type = "g2"
root@liudali-csi-amd64-node-0:~/ibm-vpc-block-csi-driver/scripts# kubectl get po -A |grep vpc-block
kube-system    ibm-vpc-block-csi-controller-bdfdf4657-m4n7v       6/6     Running            0             3m34s
kube-system    ibm-vpc-block-csi-node-9dqkd                       3/3     Running            0             3m32s
kube-system    ibm-vpc-block-csi-node-phrfb                       2/3     CrashLoopBackOff   5 (33s ago)   3m34s
root@liudali-csi-amd64-node-0:~/ibm-vpc-block-csi-driver/scripts#

there is one CrashLoopBackOff is expected.

GunaKKIBM commented 1 year ago

I am sorry, looks like I am missing something. Why is that one CrashLoopBackOff expected?

liudalibj commented 1 year ago

I am sorry, looks like I am missing something. Why is that one CrashLoopBackOff expected?

Sorry for make you confuse @GunaKKIBM I have two work nodes one arch is amd64 and another one arch is s390x, so the csi-node for s390x one failed to start is expected.

GunaKKIBM commented 1 year ago

@liudalibj

Very soon, we will be supporting multiarch images, may be in a couple of weeks and will update more on the in the docs. For now, You might need to build one and create manifest for the same for the time being.

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 6 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/issues/143#issuecomment-2015773991): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.