Status Code 401

[tommyknows]

I'm trying to get the CSI plugin working on our Rancher-launched cluster on Nutanix.

The first issue I had was https://github.com/nutanix/csi-plugin/issues/22, and according to this, I need to connect to the cluster instead of prism central. However, when trying to connect to the cluster, I get a 401 (with the exact same credentials, that are passed through to the cluster).

Further, I can verify with curl that my credentials indeed work - curl <url>/api/nutanix/v3/vms/<some uuid> --user <user>:<password> works fine on both cluster and prism central.

Excerpt from the logs:

I1029 16:11:53.914981       1 controller.go:1196] provision "default/ntnx-dataset" class "ntnx-storageclass": started
I1029 16:11:53.918058       1 controller.go:471] CreateVolumeRequest {Name:pvc-4abbc9ff-75aa-4f48-a602-f4f0bedca7ad CapacityRange:required_bytes:4294967296  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[csi.storage.k8s.io/controller-expand-secret-name:ntnx-secret csi.storage.k8s.io/controller-expand-secret-namespace:kube-system csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-publish-secret-name:ntnx-secret csi.storage.k8s.io/node-publish-secret-namespace:kube-system csi.storage.k8s.io/provisioner-secret-name:ntnx-secret csi.storage.k8s.io/provisioner-secret-namespace:kube-system dataServiceEndPoint:<ip>:3260 storageContainer:SelfServiceContainer storageType:NutanixVolumes] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1029 16:11:53.918282       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ntnx-dataset", UID:"4abbc9ff-75aa-4f48-a602-f4f0bedca7ad", APIVersion:"v1", ResourceVersion:"662211", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ntnx-dataset"
I1029 16:11:53.919889       1 connection.go:180] GRPC call: /csi.v1.Controller/CreateVolume
I1029 16:11:53.919903       1 connection.go:181] GRPC request: {"capacity_range":{"required_bytes":4294967296},"name":"pvc-4abbc9ff-75aa-4f48-a602-f4f0bedca7ad","parameters":{"dataServiceEndPoint":"<ip>:3260","storageContainer":"SelfServiceContainer","storageType":"NutanixVolumes"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I1029 16:11:54.094675       1 connection.go:183] GRPC response: {}
I1029 16:11:54.095238       1 connection.go:184] GRPC error: rpc error: code = Unknown desc = NutanixVolumes: failed to create volume: NutanixVolumes: err: Failed to get storage container list from nutanix cluster, error: Max retries done: Error with request, got status code 401
I1029 16:11:54.095310       1 controller.go:979] Final error received, removing PVC 4abbc9ff-75aa-4f48-a602-f4f0bedca7ad from claims in progress
W1029 16:11:54.095323       1 controller.go:886] Retrying syncing claim "4abbc9ff-75aa-4f48-a602-f4f0bedca7ad", failure 6
E1029 16:11:54.095363       1 controller.go:908] error syncing claim "4abbc9ff-75aa-4f48-a602-f4f0bedca7ad": failed to provision volume with StorageClass "ntnx-storageclass": rpc error: code = Unknown desc = NutanixVolumes: failed to create volume: NutanixVolumes: err: Failed to get storage container list from nutanix cluster, error: Max retries done: Error with request, got status code 401
I1029 16:11:54.095474       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ntnx-dataset", UID:"4abbc9ff-75aa-4f48-a602-f4f0bedca7ad", APIVersion:"v1", ResourceVersion:"662211", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ntnx-storageclass": rpc error: code = Unknown desc = NutanixVolumes: failed to create volume: NutanixVolumes: err: Failed to get storage container list from nutanix cluster, error: Max retries done: Error with request, got status code 401

When trying to connect to Prism Central, I get the same issue as described in #22.

[subodh01]

Have you created the secret as described in https://github.com/nutanix/csi-plugin/blob/master/example/ABS/ntnx-secret.yaml ? Please decode the secret data and check prism-ip, port, user and password. Also confirm that user/password work with Prism UI.

[tommyknows]

Already did, the secret is perfectly fine. I also tried the UI (which works) and curl, which works too.

I had this issue once before on a different project with the https://github.com/nutanix/docker-machine driver, which seems to use the v0.8 API which is disabled or something on our Cluster. There I got a 401 too when trying to connect. I fixed it by updating the API that is being used to v3, which also enabled me to use PC instead of the cluster.

[tommyknows]

BTW: would it be possible to open-source the CSI Plugin? (instead of just open-sourcing the deployment manifests)

It would make it easier to debug issues like that ourselves 🙂

[subodh01]

Here are the things that you can do.

1. You can reach out to nutanix support for faster resolution of the issue.

2. You can try looking at /home/nutanix/data/logs/prism_gateway.log for clues on all CVMs of nutanix cluster.

3. Try https://www.base64encode.org/ for base64 encoding. linux command line may have trouble handling special chars in password.

[tommyknows]

The issue was within our cluster - we didn't allow "Local" users to log in. Interestingly, it's still possible to log in to the v3 API nonetheless, the provisioner just seems to use v1.