MountVolume.SetUp failed for volume "xxx": invalid character '\r' in string literal

[sukruthakittur]

Describe the bug MountVolume.SetUp failed for volume "xxx": invalid character '\r' in string literal

Steps To Reproduce

1. Add SPN secret to kube - kubectl create secret generic kvcreds --from-literal clientid=CLIENTID --from-literal clientsecret=CLIENTSECRET --type=azure/kv

2. Deploy yml kubectl create -f deployment/nginx-flex-kv.yaml

Here is my deployment yaml. I also modified the nginx-flex-kv yml file to match the below settings.

spec:
      containers:
      - name: nginx-flexkv
        image: nginx
        volumeMounts:
        - name: kv
          mountPath: /kvmnt
          readOnly: true
      volumes:
      - name: kv
        flexVolume:
          driver: "azure/kv"
          secretRef:
            name: eecreds
          options: 
            usepodidentity: "false"
            keyvaultname: "my-vault"
            keyvaultobjectnames: test          
            keyvaultobjecttypes: key            
            resourcegroup: "my-group"       
            subscriptionid: "subscriptionid"        
            tenantid: "tenantid"    

Expected behavior Should mount the volume

Key Vault FlexVolume version Latest

Access mode: service principal or pod identity Service Principal

Kubernetes version Client Version: v1.13.2 Server Version: v1.13.7

Additional context I am not sure if there is something wrong with my yml file. I validated the deployment yml and nginx-flex-kv.yml and they seem to be valid yaml files.

[ritazh]

@sukruthakittur Thanks for reporting this issue. Few things to help us debug:

1. Can you please share the specific errors you got from kubectl describe pod?

2. Can you pls share logs at /var/log/kv-driver.log from the agent node running your pod?

3. This might just be a typo but your deployment yaml is referencing the eecreds k8s secret

   secretRef:
   name: eecreds

   But the secret you created above is kubectl create secret generic kvcreds

4. Please make sure your SP has read access to your key vault instance as well as the objects in the key vault.

[sukruthakittur]

1. Events:
   Type     Reason       Age                   From                               Message
   ----     ------       ----                  ----                               -------
   Normal   Scheduled    47m                   default-scheduler                  Successfully assigned default/nginx-flex-kv to aks-agentpool-74118896-4
   Warning  FailedMount  16m (x23 over 47m)    kubelet, aks-agentpool-74118896-4  MountVolume.SetUp failed for volume "eekv" : invalid character '\r' in string literal
   Warning  FailedMount  2m16s (x20 over 45m)  kubelet, aks-agentpool-74118896-4  Unable to mount volumes for pod "nginx-flex-kv_default(c923289c-c36a-11e9-961c-0a58ac1f032d)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx-flex-kv". list of unmounted volumes=[eekv]. list of unattached volumes=[eekv default-token-m7pn2]

2. Failed to unmarshal output for command: mount, output: "{\"status\": \"Failure\", \"message\": \"/etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0820 18:39:55.044786 127431 main.go:82] [error] : failed to get vault: failed to get vault ee-bc-tax-vault: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/3d66dc2c-7be9-474a-89fe-1a00b0511b0e/resourceGroups/a3tsmidrsg03/providers/Microsoft.KeyVault/vaults/ee-bc-tax-vault?api-version=2016-10-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {error:invalid_client,error_description:AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: fca2375e-5275-4881-8345-cc252f446a00\r\nCorrelation ID: 99f34c20-446a-4a1b-bd49-f248476ecbef\r\nTimestamp: 2019-08-20 18:39:55Z,error_codes:[7000215],timestamp:2019-08-20 18:39:55Z,trace_id:fca2375e-5275-4881-8345-cc252f446a00,correlation_id:99f34c20-446a-4a1b-bd49-f248476ecbef,error_uri:https://login.microsoftonline.com/error?code=7000215} \"}\n", error: invalid character '\r' in string literal
   W0820 18:39:55.071712    4489 driver-call.go:150] FlexVolume: driver call failed: executable: /etc/kubernetes/volumeplugins/azure~kv/kv, args: [mount /var/lib/kubelet/pods/c923289c-c36a-11e9-961c-0a58ac1f032d/volumes/azure~kv/eekv {"keyvaultname":"ee-bc-tax-vault","keyvaultobjectnames":"test","keyvaultobjecttypes":"key","keyvaultobjectversions":"","kubernetes.io/fsType":"","kubernetes.io/pod.name":"nginx-flex-kv","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"c923289c-c36a-11e9-961c-0a58ac1f032d","kubernetes.io/pvOrVolumeName":"eekv","kubernetes.io/readwrite":"rw","kubernetes.io/secret/clientid":"NTkxMDk0MmYtYmZhYy00ZDMwLTkyNjAtYWQ3ODg2ZGNmYTYy","kubernetes.io/secret/clientsecret":"NTkxMDk0MmYtYmZhYy00ZDMwLTkyNjAtYWQ3ODg2ZGNmYTYy","kubernetes.io/serviceAccount.name":"default","resourcegroup":"","subscriptionid":"","tenantid":"","usepodidentity":"false"}], error: exit status 1, output: "{\"status\": \"Failure\", \"message\": \"/etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0820 18:39:55.044786 127431 main.go:82] [error] : failed to get vault: failed to get vault ee-bc-tax-vault: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/subid/resourceGroups/groupid/providers/Microsoft.KeyVault/vaults/ee-bc-tax-vault?api-version=2016-10-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {error:invalid_client,error_description:AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: fca2375e-5275-4881-8345-cc252f446a00\r\nCorrelation ID: 99f34c20-446a-4a1b-bd49-f248476ecbef\r\nTimestamp: 2019-08-20 18:39:55Z,error_codes:[7000215],timestamp:2019-08-20 18:39:55Z,trace_id:fca2375e-5275-4881-8345-cc252f446a00,correlation_id:99f34c20-446a-4a1b-bd49-f248476ecbef,error_uri:https://login.microsoftonline.com/error?code=7000215} \"}\n"

3. Yes, that was a typo.

4. From the logs looks like your are right, there is some permission issues with my SPN.?

[ritazh]

Invalid client secret is provided

Can you pls make sure the client secret is valid?

[sukruthakittur]

Yep, thank you. I had the wrong secret.

[mohamedfasil]

@ritazh I'm also having the same issue, How to view the logs at /var/log/kv-driver.log from the agent node running the pod?

[aramase]

@mohamedfasil If you are using aks, you should be able to ssh into your agent node (https://docs.microsoft.com/en-us/azure/aks/ssh) to view the logs.