Ephemeral OS support on temporary disk (DiffDiskPlacement)

[george-zubrienko]

What happened: While creating a nodepool for AKS 1.21.2 that uses E4ds_v5 machines with Ephemeral OS, via terraform (provider 2.86), getting an error

The virtual machine size Standard_E4ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes.

Documentation for Eds_v5 says they support Ephemeral OS:

https://docs.microsoft.com/en-us/azure/virtual-machines/edv5-edsv5-series

What you expected to happen: Nodepool provisioned successfully

How to reproduce it (as minimally and precisely as possible): Create an AKS cluster via terraform, add a azurerm_kubernetes_cluster_node_pool resource with vm_size = Standard_E4ds_v5 and os_disk_type = "Ephemeral"

Anything else we need to know?: -

Environment:

• Kubernetes version (use kubectl version): 1.21.2
• Size of cluster (how many worker nodes are in the cluster?): new nodepool

[ghost]

Hi george-zubrienko, AKS bot here :wave: Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such: 1) If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster. 2) Please abide by the AKS repo Guidelines and Code of Conduct. 3) If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics? 4) Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS. 5) Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue. 6) If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[JoniRad]

I encountered this issue as well with a Dads_v5 SKU and I think I know why. These SKUs have a temp/resource disk, but no cache disk. Virtual Machines support ephemeral OS disk storage on both cache and temp disks, which is why the VM docs say Ephemeral OS disks are supported on these SKUs.

However, support for storage on temp disks was added after support for storage on cache disks and it appears AKS at the moment only supports storage on cache disks, which these SKUs do not have. Also see https://github.com/Azure/AKS/issues/2068#issuecomment-775497175 and https://github.com/Azure/AKS/issues/2003#issuecomment-738184512

@alexeldeib are there still plans to support Ephemeral OS disk storage on temp disks in AKS?

[alexeldeib]

that analysis is correct (nice digging), and yes we will support it. it only GA'd fairly recently, which was the blocker. I'd already done some work to support it but we were waiting for the GA of the functionality to allow it in AKS.

I don't have an ETA, but the work is fairly straightforward (mostly tweaking our user-facing validation). depending on holiday schedules might be able to get this released before the end of the year. I can circle back here when I've done the work and have a better ETA for a release in AKS.

[alexeldeib]

As per this doc, dads_v5 should support both.

Where do you see that? I see no mention of dads_v5 in that doc. It only mentions DiffDiskPlacement generically -- not all VM sizes have both temp disk and cache.

Looking over here, I don't see mention of a cache disk with dads_v5: https://docs.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series

And checking az vm list-skus -r virtualMachines, which is the API AKS backend will validate against, I also see no mention of CacheDiskBytes, only ResourceVolumeMB (aka temp disk).

From AKS perspective, we aren't planning to directly expose DiffDiskPlacement right now. Whatever size you give us, if it fits in exactly one of os cache or temp disk, we will put it there. If it fits in either, we will prefer using the os cache to avoid taking the temporary disk away from users for other scenarios.

{
    "apiVersions": null,
    "capabilities": [
      {
        "name": "MaxResourceVolumeMB",
        "value": "614400"
      },
      {
        "name": "OSVhdSizeMB",
        "value": "1047552"
      },
      {
        "name": "vCPUs",
        "value": "16"
      },
      {
        "name": "MemoryPreservingMaintenanceSupported",
        "value": "True"
      },
      {
        "name": "HyperVGenerations",
        "value": "V1,V2"
      },
      {
        "name": "MemoryGB",
        "value": "64"
      },
      {
        "name": "MaxDataDiskCount",
        "value": "32"
      },
      {
        "name": "CpuArchitectureType",
        "value": "x64"
      },
      {
        "name": "LowPriorityCapable",
        "value": "True"
      },
      {
        "name": "PremiumIO",
        "value": "True"
      },
      {
        "name": "VMDeploymentTypes",
        "value": "PaaS,IaaS"
      },
      {
        "name": "vCPUsAvailable",
        "value": "16"
      },
      {
        "name": "ACUs",
        "value": "230"
      },
      {
        "name": "vCPUsPerCore",
        "value": "2"
      },
      {
        "name": "CombinedTempDiskAndCachedIOPS",
        "value": "75000"
      },
      {
        "name": "CombinedTempDiskAndCachedReadBytesPerSecond",
        "value": "1048576000"
      },
      {
        "name": "CombinedTempDiskAndCachedWriteBytesPerSecond",
        "value": "268435456"
      },
      {
        "name": "UncachedDiskIOPS",
        "value": "25600"
      },
      {
        "name": "UncachedDiskBytesPerSecond",
        "value": "402653184"
      },
      {
        "name": "EphemeralOSDiskSupported",
        "value": "True"
      },
      {
        "name": "EncryptionAtHostSupported",
        "value": "True"
      },
      {
        "name": "CapacityReservationSupported",
        "value": "False"
      },
      {
        "name": "AcceleratedNetworkingEnabled",
        "value": "True"
      },
      {
        "name": "RdmaEnabled",
        "value": "False"
      },
      {
        "name": "MaxNetworkInterfaces",
        "value": "8"
      }
    ],
    "capacity": null,
    "costs": null,
    "family": "standardDADSv5Family",
    "kind": null,
    "locationInfo": [
      {
        "location": "westeurope",
        "zoneDetails": [
          {
            "Name": [
              "2"
            ],
            "capabilities": [
              {
                "name": "UltraSSDAvailable",
                "value": "True"
              }
            ],
            "name": null
          }
        ],
        "zones": [
          "2"
        ]
      }
    ],
    "locations": [
      "westeurope"
    ],
    "name": "Standard_D16ads_v5",
    "resourceType": "virtualMachines",
    "restrictions": [],
    "size": "D16ads_v5",
    "tier": "Standard"
  },

[alexeldeib]

Also these force you to choose managed disks when I try to create a VMSS from the portal. Not sure if this will be a requirement for AKS. If yes, it is a huge bummer as we would be paying for managed disks only because the platform forces us to use it.

I don't understand this. No, you won't be forced to use a managed disk (other than the free read-only HDD you don't have access to which backs the image before it's copied to local ephemeral storage).

[alexeldeib]

will this support faster or similar I/O as existing OS cache based disks for V4 series

see above json output // documents for specific vm sizes:

      {
        "name": "CombinedTempDiskAndCachedIOPS",
        "value": "75000"
      },
      {
        "name": "CombinedTempDiskAndCachedReadBytesPerSecond",
        "value": "1048576000"
      },
      {
        "name": "CombinedTempDiskAndCachedWriteBytesPerSecond",
        "value": "268435456"
      },
      {
        "name": "UncachedDiskIOPS",
        "value": "25600"
      },
      {
        "name": "UncachedDiskBytesPerSecond",
        "value": "402653184"
      },

[sanglt]

We having similar issue with Standard_D4ds_v4 VM size.

Before Standard_D4ds_v4 have 100GB cached disk. With the recently update Azure remove the cached disk, so only Temp disk left. And AKS it not support Ephemeral OS on Temp disk yet.

As the result, our node autoscaling some time is not working as expected. I need to created another nodepool with the Managed OS disk.

So support Ephemeral OS on Temp disk is very critical for us.

[asubmani]

Where do you see that? I see no mention of dads_v5 in that doc. It only mentions DiffDiskPlacement generically -- not all VM sizes have both temp disk and cache.

I made a mistake here & deleted my post. Didn't bother to check the column names (i.e. uncached).. have been using Das_v4 for a while and used to referring the number in parenthesis (). i.e. Assumed that all disks will support caching (esp new ones) and misread Max uncached disk throughput: IOPS/MBps as "Max cached and temp storage throughput: IOPS / MBps (cache size in GiB)"

A few other points on "Managed Disks" In AKS terminology the Disks are either "Managed vs Ephemeral" compared to regular Azure VM terminology which differentiates a "managed disk vs Unmanaged OS disk", with managed disks being a pre-req for using ephemeral OS. If true then the AKS doc on az aks nodepool list should display the osDiskType field as either "Persistent vs Ephemeral" instead of "Managed vs Ephemeral"

And checking az vm list-skus -r virtualMachines, which is the API AKS backend will validate against, I also see no mention of CacheDiskBytes, only ResourceVolumeMB (aka temp disk).

it seems

1) All disks in AKS VMMSS are "Managed disks anyways" if using current supported AKS versions. https://docs.microsoft.com/en-us/azure/virtual-machines/ephemeral-os-disks

2) Still not clear if we can benefit from the dadsv5 series without compromising disk IO. If I go from Standard_D8as_v4 with 200 Gb "Ephemeral OS" disk size to ---> Standard_D8ads_v5 with 200 Gb temp disk, what will happen to my IOPS?

If the Temp disks are free in the v5's then I would probably end up spinning up the max allowed size for a temp VM.

[asubmani]

From AKS perspective, we aren't planning to directly expose DiffDiskPlacement right now. Whatever size you give us, if it fits in exactly one of os cache or temp disk, we will put it there. If it fits in either, we will prefer using the os cache to avoid taking the temporary disk away from users for other scenarios.

If it fits either; and is automatic, we will wait for AKS to allow this (probably right after the holidays). Seems the below command snip should work with AKS determining that temp disk is what needs to be used as OS cache is not available.

az aks nodepool add -s Standard_D8ads_v5 --node-osdisk-type Ephemeral --node-osdisk-size 300

[mrowken]

We're in the 2022 - is there any progress on this issue as it is very awaited to unlock our upgrade plans ?

[mkemmerz]

Same issue for us with D8ds_v5. We would really like to use a v5. Now we had to go back a older v3 because of this issue.

[george-zubrienko]

@alexeldeib Do you have any updates on this one? Should we try to create a nodepool again or no progress so far?

In general, it would be nice to have an understanding when a new VM generation usually becomes "fully supported" by AKS. So far v5 has been in preview for a while, then it was released, but it doesn't feel like we are okay to use those in AKS clusters.

[phealy]

@george-zubrienko The problem isn't that the VM generation isn't fully supported by AKS; the problem is that AKS doesn't currently expose a way for you to specify the DiffDiskPlacement argument that will tell VMSS to use your temporary disk for ephemeral OS disk. Because of that, you can currently only use AKS Ephemeral OS disk on VM SKUs that have cache.

We are working on this right now - we actually had a lengthy conversation about how to implement this in a simple but future proof way earlier this week.

[sanglt]

@phealy The issue is, I can't launch the v5 instances with AKS with normal OS disk too:

Message="Virtual Machine size: 'Standard_D8ads_v5' is not supported for subscription xxxxxx in location 'australiaeast'

[phealy]

@phealy The issue is, I can't launch the v5 instances with AKS with normal OS disk too:

Message="Virtual Machine size: 'Standard_D8ads_v5' is not supported for subscription xxxxxx in location 'australiaeast'

That's not AKS - that VM SKU isn't available in that region at this time. You can check with az vm list-skus -l australiaeast.

[george-zubrienko]

@george-zubrienko The problem isn't that the VM generation isn't fully supported by AKS; the problem is that AKS doesn't currently expose a way for you to specify the DiffDiskPlacement argument that will tell VMSS to use your temporary disk for ephemeral OS disk. Because of that, you can currently only use AKS Ephemeral OS disk on VM SKUs that have cache.

@phealy thank you for the update. That's why I wrote 'fully supported'. Ephemeral disk is one of VM 'features' that we use extensively, and in my mind, if those features are not supported by AKS, then using such VMs is out of question. I mentioned earlier, we can totally live without windows containers, devops integrations, ingress gateways, but when a new VM generation goes live, we would like to have it available for use in AKS. AKS is a decent product, just us nerds want more focus on low-level stuff, rather than quality of life features :) Thank you again for all the work you are doing! Can't wait to use v5 :)

[alexeldeib]

if those features are not supported by AKS, then using such VMs is out of question.

the v5 sku you are trying to use was released 4 months before ephemeral os on temp disk was GA. we supported the skus as they existed, then new features were added ;)

[alexeldeib]

initial functionality should get picked up in 2/3 rp release, can take a week or two after that to be fully deployed to public + sov cloud regions.

this just loosens the validation for ephemeral os and allows cache/temp disk with a preference to cache disk but doesn't yet allow selecting a placement via API when both are possible in a given config.

[george-zubrienko]

Awesome! I'll try next Monday on westeurope and circle back with results.

[asubmani]

initial functionality should get picked up in 2/3 rp release, can take a week or two after that to be fully deployed to public + sov cloud regions.

this just loosens the validation for ephemeral os and allows cache/temp disk with a preference to cache disk but doesn't yet allow selecting a placement via API when both are possible in a given config.

Does this mean az aks nodepool add -s Standard_D8ads_v5 --node-osdisk-size 512 will automatically create a "DATA" disk of size 512 GB and treat it as epehmeral?

[alexeldeib]

No, there are no data disks involved. az aks nodepool add -s Standard_D8ads_v5 --node-osdisk-size **256** would result in ephemeral os on temp disk since the VM size has 300GB of temp disk but does not have a cache (as in your earlier comment)

[george-zubrienko]

Well, no luck today @alexeldeib . Again in a week? We are on westeurope.

azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"]: Creating...
╷
│ Error: creating/updating Managed Kubernetes Cluster Node Pool "spot5" (Resource Group "esd-spark-test"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VMCacheSizeTooSmall" Message="The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS."
│ 
│   with azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"],
│   on aks.tf line 105, in resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool":
│  105: resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool" {
│ 
╵

[asubmani]

No, there are no data disks involved. az aks nodepool add -s Standard_D8ads_v5 --node-osdisk-size **256** would result in ephemeral os on temp disk since the VM size has 300GB of temp disk but does not have a cache (as in your earlier comment)

Thanks. looks like we will be using dasd4 series for a while. Ephemeral OS (non-remote storage) is important for us as it offers the best disk I/O.

As of now, there are no AMD EPYC based _v5 available with ephemeral OS. Seems like if we go with a _V5 only Intel based CPU's are the only available option in "General compute" D series.

Not an AKS issue but unfortunately we have to stick to an older generation AMD (which will probably be faster than what Intel will come up with for the next year) :)

I opened a feedback here to add support for Ephemeral OS https://feedback.azure.com/d365community/idea/ad25dbc1-3e88-ec11-a81b-6045bd7956bb

[alexeldeib]

Well, no luck today

Yeah, release is a bit slow. It only got started on 2/4 and looks like hasn't made a ton of progress. give it a little time.

[george-zubrienko]

Well, same story again in westeurope. I see issue is moved to GA, but then what's wrong? @alexeldeib

azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"]: Creating...
╷
│ Error: creating/updating Managed Kubernetes Cluster Node Pool "spot5" (Resource Group "esd-spark-test"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VMCacheSizeTooSmall" Message="The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 137438953472 bytes. Use a VM size with larger cache or disable ephemeral OS."
│ 
│   with azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"],
│   on aks.tf line 105, in resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool":
│  105: resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool" {
│ 
╵

Just in case you noticed requested size here is more than the size of a temp disk on E2ds vm, tried with 64gb:

containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VMCacheSizeTooSmall" Message="The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS."
│ 
│   with azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"],
│   on aks.tf line 105, in resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool":
│  105: resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool" {
│ 
╵

Also tried updating terraform provider to latest 2.96.0, no luck.

UPD. Tried today with just az cli and result is full of sadness:

xami@DK6313:~$ az aks nodepool add -g esd-spark-test -n nodepool2 --cluster-name esd-spark-test --node-osdisk-type Ephemeral --node-osdisk-size 64 -s Standard_E2ds_v5
(VMCacheSizeTooSmall) The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS.
Code: VMCacheSizeTooSmall
Message: The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS.

[kevinharing]

Well, same story again in westeurope. I see issue is moved to GA, but then what's wrong? @alexeldeib

azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"]: Creating...
╷
│ Error: creating/updating Managed Kubernetes Cluster Node Pool "spot5" (Resource Group "esd-spark-test"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VMCacheSizeTooSmall" Message="The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 137438953472 bytes. Use a VM size with larger cache or disable ephemeral OS."
│ 
│   with azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"],
│   on aks.tf line 105, in resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool":
│  105: resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool" {
│ 
╵

Just in case you noticed requested size here is more than the size of a temp disk on E2ds vm, tried with 64gb:

containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="VMCacheSizeTooSmall" Message="The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS."
│ 
│   with azurerm_kubernetes_cluster_node_pool.spark_cluster_worker_pool["spark-swarm-5"],
│   on aks.tf line 105, in resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool":
│  105: resource "azurerm_kubernetes_cluster_node_pool" "spark_cluster_worker_pool" {
│ 
╵

Also tried updating terraform provider to latest 2.96.0, no luck.

UPD. Tried today with just az cli and result is full of sadness:

xami@DK6313:~$ az aks nodepool add -g esd-spark-test -n nodepool2 --cluster-name esd-spark-test --node-osdisk-type Ephemeral --node-osdisk-size 64 -s Standard_E2ds_v5
(VMCacheSizeTooSmall) The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS.
Code: VMCacheSizeTooSmall
Message: The virtual machine size Standard_E2ds_v5 has a cache size of 0 bytes, but the OS disk requires 68719476736 bytes. Use a VM size with larger cache or disable ephemeral OS.

Rollout of release is not yet finished. Is expected to conclude today, see release notes.

@alexeldeib I'm wondering why it's listed as a preview feature (in the release notes), can anyone shed a light on that?

[george-zubrienko]

Massive!

xami@DK6313:~$ az aks nodepool add -g esd-spark-test -n nodepool2 --cluster-name esd-spark-test --node-osdisk-type Ephemeral --node-osdisk-size 64 -s Standard_E2ds_v5
 / Running ..

Works! Final probes today and I think we are good to close this one :)

[george-zubrienko]

Our AKS cluster with 180 new E4ds/E8ds with ephemeral os disk running flawlessly. As I see it, issue is resolved, thank you so much for the commitment and all communications that made tracking this nice and easy!

[davem-git]

whats the timeline to get v5 added to eastasia?

[alexeldeib]

Our AKS cluster with 180 new E4ds/E8ds with ephemeral os disk running flawlessly. As I see it, issue is resolved, thank you so much for the commitment and all communications that made tracking this nice and easy!

Glad to hear it :) appreciate your patience!

whats the timeline to get v5 added to eastasia?

Unfortunately AKS doesn't control this, it's up to underlying compute platform. Support ticket might be able to get you a better answer. In the general case AKS programmatically onboards new SKUs as they become available via API, so it should be immediately available on AKS once it's available in eastasia generally.

[asubmani]

Are these disks charged extra? With Ephemeral OS disks in cached mode, these disks were at 0 cost and made them a no-brainer to adopt for AKS. Will there be an additional charge to the DISK on top of the VM now with the new disks (non-cached mode). Also are there any benchmarks between cached mode ephemeral vs non-cached? looking for something similar to https://docs.microsoft.com/en-us/azure/virtual-machines/windows/compute-benchmark-scores

[alexeldeib]

Are these disks charged extra?

no

are there any benchmarks between cached mode ephemeral vs non-cached?

not that AKS publishes officially. I'd suggest using fio or similar with a representative workload for your app. You may have luck googling blogs trying this in the past.