Container app tries to mount NFS file share as CIFS share

robbelouwet commented 9 months ago

This issue is a: (mark with an x)

[x] bug report -> please search issues before submitting
[ ] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description

When deploying a container app and mounting an NFS Azure FIle share, the mount operation fails with mount error(13): Permission denied. The mount operation uses arguments-t cifs, which leads me to believe it's trying to mount an CIFS share instead of an NFS share.

Steps to reproduce

Deploy a vnet with a default /24 subnet and a second /23 subnet.
Deploy a Premium tier storage account type with account type: 'File Shares'.
In the storage account, create a NFS file share with 'No Root Squash' enabled.
Disable 'Secure Transfer Required' and enable a private endpoint on the storage account in the 'default' subnet as it suggests in order to enable NFS file shares.
Create a new container app with TCP and ingress enabled (image doesn't matter), with a consumption-only container app environment using a custom vnet, dedicate the /23 subnet for the container apps environment infrastructure.
Go to the container app environment > Settings > File Share, add a new File Share storage definition and add the storage account name, key and File Share name of the NFS File Share you just created, set the access mode to 'Read/Write'.
Go to the container app, go to 'Revisions' > 'Create new revision'. Now click on the existing container image such that the side menu pops open on the right, now select the 'Volume Mounts' tab, expand 'File Shares' and add a File Share. The name of the Storage Definition created in step 3 should be present in the drop-down, select this. Choose any mount path (doesn't matter which one).
The new revision will reach 'Activation Failed' status because it fails to mount. Go to 'Monitoring' > 'Log Stream' and toggle 'System' logs. You now see the error messages.

Expected behavior [What you expected to happen.] Container App reaches 'Running' status and is able to mount the NFS File Share.

Actual behavior [What actually happened.] Container App gets stuck in 'Activating' state because the container is unable to mount the file share. After a while, the running status reaches 'Activtion Failed'.

Screenshots

{
        "TimeStamp": "2023-12-16 12:49:17 \u002B0000 UTC",
        "Type": "Warning",
        "ContainerAppName": "papermc-dev-server",
        "RevisionName": "papermc-dev-server--eycpdk3",
        "ReplicaName": "papermc-dev-server--eycpdk3-8565d55b84-j5lkq",
        "Msg": "MountVolume.SetUp failed for volume \u0022papermc-dev-share\u0022 : rpc error: code = Internal desc = volume(csi-0be001dc35a21f8fb32927dd80832710e3a999738270c4aec37f464bcc69bda4) mount //papermcdevstorage.file.core.windows.net/papermc-dev-share on /var/lib/kubelet/pods/903eb86e-384b-46ae-83fa-990e5bad471e/volumes/kubernetes.io~csi/papermc-dev-share/mount failed with mount failed: exit status 32\nMounting command: mount\nMounting arguments: -t cifs -o ,dir_mode=0777,actimeo=30,mfsymlinks,nosharesock,file_mode=0777,\u003Cmasked\u003E //papermcdevstorage.file.core.windows.net/papermc-dev-share /var/lib/kubelet/pods/903eb86e-384b-46ae-83fa-990e5bad471e/volumes/kubernetes.io~csi/papermc-dev-share/mount\nOutput: mount error(13): Permission denied\nRefer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)\n\nPlease refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.",
        "Reason": "FailedMount",
        "EventSource": "ContainerAppController",
        "Count": 1
    }

Additional context

N/A

howang-ms commented 9 months ago

The behavior is expected for now. The Azure NFS File Share mount in the Azure Container App will be supported in Q1 2024.

robbelouwet commented 7 months ago

Hi there! Any idea on a specific ETA of this feature?

Thanks!

anthonychu commented 7 months ago

This is currently available in the 2023-11-02-preview API version in ARM and Bicep, and using the preview containerapp CLI extension.

We're still waiting to add support for it in the Azure portal. In the meantime, you can test it out by ensuring you have the latest Azure CLI installed, as well as 0.3.47 or later of the containerapp extension.

When configuring the file share in the Container Apps environment, use az containerapp env storage set -g MyResourceGroup -n MyEnv --storage-name MyStorageName --storage-type NfsAzureFile --access-mode ReadOnly --server MyNfsServer.file.core.windows.net --file-share /MyNfsServer/MyShareName Note that --server takes the full FQDN of the server. --file-share is just the server name and share name, following the format /MyNfsServer/MyShareName exactly.
When creating the volume, use NfsAzureFile instead of AzureFile.

NFS file shares require that you have a Container Apps environment with a custom VNET, and that it's been connected to the storage account.

We expect to officially announce the feature in public preview in the next 2-3 months.

abij commented 7 months ago

Thx @anthonychu for the preview, looking forward to this feature! I assume this is under heavy development, the azure portal does not display this type correctly. But we are able to create such a share using your example. We use the ServiceEndpoint instead of the private-endpoint connectivity.

currently: az containerapp env storage list -g <group> -n <name> throws in internal server error afterwards though.

How to you specify the credentials (account-key) for this share? In the portal under shares? update o, found it, there is a flag --azure-file-account-key / -k option.

Do you know with mount options are suitable? vers=4,minorversion=1,sec=sys,nconnect=4 I found on the share itself.

xstrama commented 3 months ago

This is currently available in the 2023-11-02-preview API version in ARM and Bicep, and using the preview containerapp CLI extension.

We're still waiting to add support for it in the Azure portal. In the meantime, you can test it out by ensuring you have the latest Azure CLI installed, as well as 0.3.47 or later of the containerapp extension.

When configuring the file share in the Container Apps environment, use az containerapp env storage set -g MyResourceGroup -n MyEnv --storage-name MyStorageName --storage-type NfsAzureFile --access-mode ReadOnly --server MyNfsServer.file.core.windows.net --file-share /MyNfsServer/MyShareName Note that --server takes the full FQDN of the server. --file-share is just the server name and share name, following the format /MyNfsServer/MyShareName exactly.

When creating the volume, use NfsAzureFile instead of AzureFile.

NFS file shares require that you have a Container Apps environment with a custom VNET, and that it's been connected to the storage account.

We expect to officially announce the feature in public preview in the next 2-3 months.

Could you provide an example in Bicep for this, preferably with a private endpoint?

I have tried with both Azure CLI and Bicep but the container always end up in a degraded state with no error messages. The format of server and shareName when using Azure CLI is documented but when it comes to Bicep it's not, therefor an example would help. I looked at the ARM template and ended up with a string interpolation expression to format the path for server and serverName. @anthonychu

microsoft / azure-container-apps