[BLOCKER] Listing EC2 disks is painful or impossible

[jzabroski]

See: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-windows-volumes.html#windows-disks - note there is no built-in way to do this, e.g. with Get-Disk on windows

Also note that I don't think this works on Windows Server 2016, as Get-Disk will never return dynamic disks, and on my r5d instance, the ec2 gp2 volumes are always dynamic disks. They only show up using the legacy Disk Management GUI utility, and don't even show up as disks in Server Manager GUI utility.

and my Serverfault thread regarding this:

https://serverfault.com/questions/963392/why-is-listing-ec2-disks-so-painful-on-windows

[jzabroski]

The root cause of this problem is two-fold:

1. Get-Disk does not return dynamic disks, because dyanmic disks are deprecated https://blogs.technet.microsoft.com/tip_of_the_day/2014/04/25/tip-of-the-day-dynamic-disks-and-windows-powershell/
2. On Windows 8 and Windows Server 2012, Microsoft deprecated LDM in favor of Storage Spaces. https://en.wikipedia.org/wiki/Logical_Disk_Manager

Separately, Dynamic disks don't play well with Linux. http://blog.itekako.com/technical/2017/02/23/short-lived-life-of-dynamic-disks/

[matteo-prosperi]

Hello John, EC2 support will be in a better position to help you. This repo is only related to the AWSPowerShell module which is meant to interact with AWS services (for example creating new EC2 instances and configure them), not to work with your local EC2 instance. So I would suggest you to creat a case here (https://console.aws.amazon.com/support/home).

I can still provide you a couple of suggestions that may help you out.

1. I tried attaching a new gp2 disk to an EC2 instance (Windows Server 2019) and it was not initialized at first. When I initialized it became a Basic disk. In order to get a Dynamic disk I had to explicitly convert it to Dynamic.
2. You could try to get the list of the disks through WMI doing something like Get-WmiObject -Query "SELECT * FROM Win32_PhysicalMedia. This returns all physical disks, even dynamic ones.
3. If you are interested in your mounted drives instead of the physical disks, you could try something like [System.IO.DriveInfo]::GetDrives().

[jzabroski]

@matteo-prosperi How did you explicitly convert it to dynamic?

I'm pretty sure I'm not sophisticated enough to have explicitly done that on purpose. Is it possible EC2 Console did this when I modified an instance by expanding the volume? e.g., if the backplane had no available space, maybe it 'optimized' the instance such that the storage now spans multiple disks. I will note that this was on Windows Server 2016, and you couldn't reproduce it on Windows Server 2019.

I'm not interested in my mounted drives, I'm interested in automating auto-extending a drive with only one partition so that partition takes up the full drive space available. You can see my script here: https://stackoverflow.com/questions/41976411/remotely-extend-a-partition-using-wmi/48998751#48998751

I think it's non-optimal to close this, although I did file an issue with EC2 Support last night as well. Aren't you in charge of the documentation related to EC2? Or don't you talk to the people who maintain it?

[jzabroski]

Also, when you say you initialized the disk, how did you do that? Below is my automated script.

Import-Module Storage;

function Format-Drives
{
    # See https://stackoverflow.com/a/42621174/1040437 (Formatting a disk using PowerShell without prompting for confirmation)
    $currentconfirm = $ConfirmPreference
    $ConfirmPreference = 'none'

    Get-Disk | Where isOffline | Set-Disk -isOffline $false
    # The next line of this script is (almost) copy-pasted verbatim from: https://blogs.technet.microsoft.com/heyscriptingguy/2013/05/29/use-powershell-to-initialize-raw-disks-and-to-partition-and-format-volumes/
    Get-Disk | Where partitionstyle -eq 'raw' | Initialize-Disk -PartitionStyle MBR -Confirm:$false -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize -IsActive | Format-Volume -FileSystem NTFS -Confirm:$false

    # See https://stackoverflow.com/a/42621174/1040437 (Formatting a disk using PowerShell without prompting for confirmation)
    $ConfirmPreference = $currentconfirm
}

Format-Drives

[matteo-prosperi]

Hi, I tried your PS function on a gp2 volume attached to a Windows Server 2016 EC2 instance. It initializes the disk as Basic and creates a partition. I then resized the volume from the AWS Console (EC2 web site) and it resulted in a larger disk with empty space. I finally resized the partition from Disk Management to use the full disk. During the whole process the disk always stayed Basic. I don't know why your disk ended up being dynamic but EC2 support may be able to help you with that.

EC2 support can help you with your needs related to configuration of EC2 instances and volumes. You can also discuss with them issues related to the EC2 documentation not being clear. Support engineers are in the best position to involve multiple teams and coordinate the resolution of customer problems.

I am closing the issue here because it is not related to the AWSPowerShell module. If you are a user of AWSPowerShell, feel free to open issues here related to bugs or feature requests.

[jzabroski]

Thanks a ton. One last question - was your EC2 instance an r5d? I wonder if the problem is only related to such instances with ephemeral storage, since ephemeral storage disappeared as an option in AWS EC2 for awhile (it wasn't possible in r3 and 4 instances as I recall), and is now back in r5d.

[matteo-prosperi]

Happy to help. For the last test I wrote you about (attach a volume, initialize and resize), I used an r5d.

[jzabroski]

I think you're right... someone must have converted it to a dynamic disk while I was on vacation (or I need slimmer fingers/better mouse because the only way I can see that a disk can become dynamic is if you explicitly convert it). I checked all my other instances and they're all Basic Disks.