Mounting Block Device on AWS M1 Mac

[sanieldalib]

Hey there!

I am trying to use the feature added here: https://github.com/cirruslabs/tart/pull/611 to mount a volume on the SSD of an AWS M1 Mac to a Tart VM.

I created a volume on the SSD called Tart using

diskutil apfs addVolume disk3 APFS Tart

which appears when running diskutil list:

% diskutil list                                 
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
   2:                 Apple_APFS Container disk3         245.1 GB   disk0s2
   3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +245.1 GB   disk3
                                 Physical Store disk0s2
   1:                APFS Volume InternalDisk            10.1 GB    disk3s1
   2:                APFS Volume Preboot                 5.2 GB     disk3s2
   3:                APFS Volume Recovery                907.0 MB   disk3s3
   4:                APFS Volume Data                    701.1 MB   disk3s5
   5:                APFS Volume VM                      20.5 KB    disk3s6
   6:                APFS Volume Tart                    827.4 KB   disk3s7

I then start the Tart VM like so:

 ~ % sudo TART_HOME=~/.tart tart run testmount --disk=/dev/disk3s7
Using block device

However once the VM boots, I get the following message:

checking Disk Utility on the VM yields the following:

diskutil list on the VM yields the following:

admin@ip-192-168-64-2 ~ % diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *245.1 GB   disk0

Finally the only volume appears to be Macintosh HD:

admin@ip-192-168-64-2 ~ % ls /Volumes 
Macintosh HD

For reference, the host is running macOS 14.2.1 and the VM macOS 14.3.

I suspect something is incorrect with the volume I created, any help would be appreciated 😄

[edigaryev]

Hello Daniel 👋

I'm afraid that when using APFS, you'll need to attach the whole block device to the VM, in your case, /dev/disk3:

sudo TART_HOME=~/.tart tart run testmount --disk=/dev/disk3

Another option would be to partition the device (can be done in Disk Utility when View → Show All Devices is enabled):

This way you'll either have an additional synthesized disk (when formatting the new partition as APFS), or a mountable partition (when formatting the new partition as HFS+/etc.).

[sanieldalib]

Thanks @edigaryev! I created a partition and was able to see the device on the VM. However, the device appears in diskutil list on the VM but I am unable to mount it.

admin@ip-192-168-64-2 ~ % diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *200.0 GB   disk0

/dev/disk1 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *215.0 GB   disk1
   1:             Apple_APFS_ISC Container disk3         524.3 MB   disk1s1
   2:                 Apple_APFS Container disk4         214.5 GB   disk1s2

/dev/disk2 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +200.0 GB   disk2
                                 Physical Store disk0
   1:                APFS Volume Test                    856.1 KB   disk2s1

/dev/disk4 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +214.5 GB   disk4
                                 Physical Store disk1s2
   1:                APFS Volume Macintosh HD            10.1 GB    disk4s1
   2:              APFS Snapshot com.apple.os.update-... 10.1 GB    disk4s1s1
   3:                APFS Volume Preboot                 5.2 GB     disk4s2
   4:                APFS Volume Recovery                897.7 MB   disk4s3
   5:                APFS Volume Data                    39.8 GB    disk4s5
   6:                APFS Volume VM                      20.5 KB    disk4s6

admin@ip-192-168-64-2 ~ % diskutil mountDisk disk2
One or more volume(s) failed to mount
admin@ip-192-168-64-2 ~ % diskutil mountDisk disk2s1
One or more volume(s) failed to mount
admin@ip-192-168-64-2 ~ % diskutil mount disk2s1    
Volume on disk2s1 failed to mount

admin@ip-192-168-64-2 ~ % ls /Volumes 
Macintosh HD

Any ideas?

[fkorotkov]

I wasn't able to use a partition as well. What I tried instead is to have another EBS volume and mount it. I also checked and it seems AWS actually doesn't recommend to use SSDs since they don't guarantee their conditions.

ec2-user@ip-172-31-94-110 ~ % diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
   2:                 Apple_APFS Container disk3         245.1 GB   disk0s2
   3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +245.1 GB   disk3
                                 Physical Store disk0s2
   1:                APFS Volume InternalDisk            10.1 GB    disk3s1
   2:                APFS Volume Preboot                 5.2 GB     disk3s2
   3:                APFS Volume Recovery                907.7 MB   disk3s3
   4:                APFS Volume Data                    643.8 MB   disk3s5
   5:                APFS Volume VM                      20.5 KB    disk3s6

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *859.0 GB   disk4
   1:                        EFI EFI                     209.7 MB   disk4s1
   2:                 Apple_APFS Container disk6         858.8 GB   disk4s2

/dev/disk5 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *32.2 GB    disk5

/dev/disk6 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +858.8 GB   disk6
                                 Physical Store disk4s2
   1:                APFS Volume Macintosh HD - Data     5.6 GB     disk6s1
   2:                APFS Volume Macintosh HD            9.9 GB     disk6s2
   3:              APFS Snapshot com.apple.os.update-... 9.9 GB     disk6s2s1
   4:                APFS Volume Preboot                 5.6 GB     disk6s3
   5:                APFS Volume Recovery                820.4 MB   disk6s4
   6:                APFS Volume VM                      20.5 KB    disk6s5

Now I formatted it to APFS:

diskutil eraseDisk APFS "Tart" /dev/disk5
diskutil unmountDisk /dev/disk5

Started a VM with that extra EBS volume:

sudo TART_HOME=$PWD/.tart tart run sonoma-base --disk=/dev/disk5

Then I can see it inside the VM after brew install cirruslabs/cli/sshpass:

sshpass -p admin ssh -o "StrictHostKeyChecking no" admin@$(tart ip sonoma-base) "ls /Volumes"
Macintosh HD
Tart

So it seems that is a viable solution to mount extra EBS volumes to VMs. These volumes will have 10-15% faster IO since Tart will directly use it instead of writing to disk.img file which is getting written to the root EBS volume. Plus you can snapshot such volumes and do anything you want.

PS weird that we were not able to get partition working. Maybe there is some way and I hope we'll figure it one day.

[sanieldalib]

Are you able to get the SSD to appear using the method above? I tried and it does not appear in the VM.

[fkorotkov]

Nope, for reason it didn't work. Showed as it was successfully mounted though. But it wasn't available inside the VM. Maybe it needs to be completely formatted but I haven't tested it.

[sanieldalib]

@fkorotkov I tried formatting it as well, but no luck 😕

[edigaryev]

@sanieldalib I think I've found a quick and dirty workaround for this.

Create a disk image of the desired size on the attached SSD:

truncate -s 250GB disk.img

Mount it (no sudo/root-privileges required) into a Tart VM instead of the actual SSD block device:

tart run --disk /path/to/disk.img <VM>

On the first mount the disk will need to be initialized from within the VM, but for the rest of the VM invocations it will auto-mount just fine, without the need for the manual intervention.

[fkorotkov]

Closing since it seems we have couple of options including mounting a separate EBS volume.

[sanieldalib]

@edigaryev Thank you for the SSD disk image workaround! I just tested it and its working for me. However, my understanding is that there is a performance cost to using a disk image in place of the SSD directly.

Have you had any luck with using the SSD directly?

[sanieldalib]

Additionally do you know if using more performant disk image formats (specifically .sparsebundle) is supported?

[sanieldalib]

Hi @edigaryev @fkorotkov, just following up on this 😄

[fkorotkov]

I think it might correlate with #790. 🤔

[edigaryev]

Have you had any luck with using the SSD directly?

Yes, see https://github.com/cirruslabs/tart/issues/724#issuecomment-1923537366.

Additionally do you know if using more performant disk image formats (specifically .sparsebundle) is supported?

Yes, they work similarly to a physical device.

[sanieldalib]

@edigaryev I've implemented your suggestion of using a SSD hosted disk image, however creating a non-sparse image using

hdiutil create -size 50g -volname Tart disk

takes quite some time prior to each build, which scales with the disk image size

hdiutil create -size 50g -volname Tart disk  0.01s user 0.02s system 0% cpu 51.253 total

Meanwhile using either --type SPARSEBUNDLE or --type SPARSE only takes 1s, regardless of size.

However, neither sparse format works as expected.

--type SPARSEBUNDLE:

tart run disk-test --disk disk.sparsebundle                               
Error: Invalid disk image. The disk image format is not recognized.

--type SPARSE: Only appears as a volume of 15MB (I assume the initial size of the sparse image), but fails to grow when writing more data.

Any ideas if it is possible to use a sparse format disk image?

CC @fkorotkov

EDIT: Using truncate as suggested seems to work, but requires initializing the disk within the VM for each build, which hdiutil avoids this.

[edigaryev]

tart run disk-test --disk disk.sparsebundle

You have to mount this sparse bundle beforehand, and then pass the resulting virtual disk to Tart, for example, if the sparse bundle was mounted as /dev/disk4:

% diskutil list /dev/disk4
/dev/disk4 (disk image):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        +53.7 GB    disk4
   1:                        EFI EFI                     209.7 MB   disk4s1
   2:                 Apple_APFS Container disk5         53.5 GB    disk4s2

Then run the following commands:

diskutil unmountDisk /dev/disk4
tart run --disk /dev/disk4 <VM>