SMB Plugin Issue

[Tharun-Sabbu]

Hello,

I'm using Kubernetes v1.15.1 with windows server 2019 workers and was using v0.0.3 release of storage plugins on windows workers.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  smb-deployment
  labels:
    name:  smb-app
  namespace: windows
spec:
  replicas: 1
  selector:
    matchLabels:
      app: smb-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app:  smb-app
    spec:
      containers:
      - name: smb
        image: microsoft/nanoserver
        command: ["C:\\Windows\\System32\\cmd.exe"]
        args: ["/C", "echo", "0", ">", "c:\\d\\smb_test.txt"]
        volumeMounts:
        - name: smb-volume
          mountPath: c:/data
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
          limits:
            cpu: 200m
            memory: 500Mi
      volumes:
      - name: smb-volume
        flexVolume:
          driver: "microsoft.com/smb.cmd"
          secretRef:
            name: "smb-secret"
          options:
            # source can be in any of the following formats 
            # \\servername\share\path  (\'s will need to be escaped)
            # smb://servername/share/path
            # //servername/share/path
            source: \\10.0.0.5\sharedfolder
      nodeSelector:
        beta.kubernetes.io/os: windows
        kubernetes.io/hostname: ip-0a230df7

Here is the error I'm getting in kubelet with debug enabled

"\"flexvolume-microsoft.com/smb.cmd/58b90ea4-e8e0-4549-a71c-6855f1f8b51a-smb-volume\" (\"58b90ea4-e8e0-4549-a71c-6855f1f8b51a\")" failed. No retries permitted until 2019-09-19 04:08:01.8282223 +0000 GMT m=+997.903271801 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"smb-volume\" (UniqueName: \"flexvolume-microsoft.com/smb.cmd/58b90ea4-e8e0-4549-a71c-6855f1f8b51a-smb-volume\") pod \"smb-deployment-8585cccf96-g77pv\" (UID: \"58b90ea4-e8e0-4549-a71c-6855f1f8b51a\") : mount command failed, status: Failure, reason: Caught exception The parameter is incorrect. with stack "

Please let me know if I'm doing something wrong, thanks.

[KnicKnic]

Didn’t mean to close, clicked the wrong button.

The plugins log to the 'Application' event channel on the appropriate worker node. Get-EventLog -LogName Application -Source Kube* -Newest 50

You can get a more detailed trace of the components by enabling debug logs may contain your passwords when you do this edit C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.com~iscsi.cmd\flexvolume.ps1 and C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.com~smb.cmd\flexvolume.ps1 and change $debug_mode = $false to $debug_mode = $true

[Tharun-Sabbu]

Thanks for quick turn

Here is the log:

log: Caught exception The parameter is incorrect.  with stack
log: smbGlobal
log: \\10.0.0.5\sharedfolder
log: Administrator

log: password redacted
log: @{kubernetes.io/fsType=; kubernetes.io/pod.name=smb-deployment-8585cccf96-7xn4j; kubernetes.io/pod.namespace=windows; kubernetes.io/pod.uid=f8a7dadd-6ced-485c-b3b4-52b196d7ba00; kubernetes.io/pvOrVolumeName=smb-volume; kubernetes.io/readwrite=rw; kubernetes.io/secret/password=password redacted; kubernetes.io/secret/username=QWRtaW5pc3RyYXRvcgo=; kubernetes.io/serviceAccount.name=default; source=\\10.0.0.5\sharedfolder}
log: Make dir c:\var\lib\kubelet\pods\f8a7dadd-6ced-485c-b3b4-52b196d7ba00\volumes\microsoft.com~smb.cmd\smb-volume\..\
log: mount
log: json:  {"kubernetes.io/fsType":"","kubernetes.io/pod.name":"smb-deployment-8585cccf96-7xn4j","kubernetes.io/pod.namespace":"windows","kubernetes.io/pod.uid":"f8a7dadd-6ced-485c-b3b4-52b196d7ba00","kubernetes.io/pvOrVolumeName":"smb-volume","kubernetes.io/readwrite":"rw","kubernetes.io/secret/password":"password redacted","kubernetes.io/secret/username":"QWRtaW5pc3RyYXRvcgo=","kubernetes.io/serviceAccount.name":"default","source":"\\\\10.0.0.5\\sharedfolder"}
log: cmdline: C:\Windows\system32\cmd.exe /c c:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.com~smb.cmd\smb.cmd mount c:\var\lib\kubelet\pods\f8a7dadd-6ced-485c-b3b4-52b196d7ba00\volumes\microsoft.com~smb.cmd\smb-volume {\"kubernetes.io/fsType\":\"\",\"kubernetes.io/pod.name\":\"smb-deployment-8585cccf96-7xn4j\",\"kubernetes.io/pod.namespace\":\"windows\",\"kubernetes.io/pod.uid\":\"f8a7dadd-6ced-485c-b3b4-52b196d7ba00\",\"kubernetes.io/pvOrVolumeName\":\"smb-volume\",\"kubernetes.io/readwrite\":\"rw\",\"kubernetes.io/secret/password\":\"password redacted\",\"kubernetes.io/secret/username\":\"QWRtaW5pc3RyYXRvcgo=\",\"kubernetes.io/serviceAccount.name\":\"default\",\"source\":\"\\\\10.0.0.5\\sharedfolder\"}
log: folder: c:\var\lib\kubelet\pods\f8a7dadd-6ced-485c-b3b4-52b196d7ba00\volumes\microsoft.com~smb.cmd\smb-volume
log: command: mount

Let me know if you need further logs

[KnicKnic]

The script is mounting the smb share using New-SmbGlobalMapping and it appears that this is failing https://github.com/microsoft/K8s-Storage-Plugins/blob/e683bce3c33df06417775378d7bb5b1d4cfe92d3/flexvolume/windows/plugins/microsoft.com~smb.cmd/smb.ps1#L38-L47

I believe the line that is failing is https://github.com/microsoft/K8s-Storage-Plugins/blob/e683bce3c33df06417775378d7bb5b1d4cfe92d3/flexvolume/windows/plugins/microsoft.com~smb.cmd/smb.ps1#L46

You should be able to run all of these commands your self and try and trouble shoot them in a powershell session.

The construct credential function is defined earlier in the file https://github.com/microsoft/K8s-Storage-Plugins/blob/e683bce3c33df06417775378d7bb5b1d4cfe92d3/flexvolume/windows/plugins/microsoft.com~smb.cmd/smb.ps1#L6-L10

I did notice one issue. Your username needs to have a domain,workgroup, or somethign defined. So instead of Administrator use 10.0.0.5\Administrator . If it is not a domain or workgroup it doesn't matter what is before the '\' however there needs to be something. This should not fix your issue. It should cause "Access is denined" why you are not getting that I do not know.

The three parameters that are passed into new-smbglobalmapping are effectively logged immediately before and they seem to not be "invalid"

[Tharun-Sabbu]

Hello,

I did manually and here is the result I got on host:

New-SmbGlobalMapping  -RemotePath \\ip-redacted\samplefolder -Credential $credential -persistent $false 2>&1

Status Local Path Remote Path
------ ---------- -----------
OK                \\ip-redacted\samplefolder

I was able to run "net use" commands and mount smb file share inside a windows pod and that also seems to be working

[Tharun-Sabbu]

I ran this directly on windows node

C:\Windows\system32\cmd.exe /c c:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.com~smb.cmd\smb.cmd mount c:\var\lib\kubelet\pods\4ce239c7-2745-42e4-98d1-da079bb9108a\volumes\microsoft.com~smb.cmd\smb-volume '{\"kubernetes.io/fsType\":\"\",\"kubernetes.io/pod.name\":\"smb-deployment-65d6dd657f-bws5t\",\"kubernetes.io/pod.namespace\":\"windows\",\"kubernetes.io/pod.uid\":\"4ce239c7-2745-42e4-98d1-da079bb9108a\",\"kubernetes.io/pvOrVolumeName\":\"smb-volume\",\"kubernetes.io/readwrite\":\"rw\",\"kubernetes.io/secret/password\":\"base64-password\",\"kubernetes.io/secret/username\":\"base64-username\",\"kubernetes.io/serviceAccount.name\":\"default\",\"source\":\"\\\\ip-redacted\\samplefolder\"}'
{"message":"Caught exception The parameter is incorrect.  with stack ","status":"Failure"}

and I got: Caught exception The parameter is incorrect.

I ran: New-SmbGlobalMapping -RemotePath \ip-redacted\samplefolder -Credential $credential -persistent $false 2>&1, was able to mount with the session on the host and also ran inside the container, once the container started running and also succeeded with session.

Tried same thing with arguments on pod but couldn't able to succeed.

Update: I found the issue, though the username and password are correctly landing up in log but somehow while credentials being constructed they are messing up and causing issues for New-SmbGlobalMapping. Might be with characters, but that's my guess.

Closing the issue, thanks for the contribution!!!

[KnicKnic]

@Tharun-Sabbu

Update:

I found the issue, though the username and password are correctly landing up in log but somehow while credentials being constructed they are messing up and causing issues for New-SmbGlobalMapping. Might be with characters, but that's my guess.

Sorry for the long delay, I got back in today. Do you know if it was a specific character, or have any suggestions on how to produce a better error message, or validation to help the user to more quickly triage these problems in the future?

[Tharun-Sabbu]

Thanks for getting back, I'm not really a expert powershell script guy, my username is something like this : "ip-xx-xx-xx-xx.ec2.internal\Administrator" and password has special characters @ and $, numbers and Alphabets. I did check multiple times decoding the username and password and every-time debug messages seems to be giving me the right credentials. Not sure what could be the issue. I know it is not a right way to hard code the credentials, but couldn't make it work with the credentials I have.

Issue is definitely in this block as per my testing, I took multiple approaches but couldn't make it work:

function ConstructCredential([string]$username, $passPlain)
{
    $securePassword = ConvertTo-SecureString -String $passPlain -AsPlainText -Force
    return New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $securePassword 
}