Dasharo / dasharo-issues

The Dasharo issue tracker
https://dasharo.com/
25 stars 0 forks source link

Add RAM Disk Driver to create a RAM backed virtual Block Device #277

Open zirblazer opened 1 year ago

zirblazer commented 1 year ago

The problem you're addressing (if any) The problem is when you have too much RAM and don't know what to do with it :D

Describe the solution you'd like Adding a RAM Disk Driver that allows the Firmware to reserve a certain capacity of physical RAM to create a Block Device that is seen by the OS as a fixed Disk Drive. While it is usually possible to do this OS-side and I don't find it very useful for Linux (Due to having tmpfs and brd Driver support easily available), creating a RAM Disk from within Firmware for Windows will remove the need to having to use third party tools to do so, so it may be more convenient.

Where is the value to a user, and who might that user be? It depends on an actual use case, ease of use, and performance characteristics. Most people don't really benefit from a dumb RAM Disk if they have to do manual backup of the RAM Disk contents, but it was recommended for things like browser Cache or other stuff that may actually benefit from being volatile. A Firmware-side RAM Disk seems convenient.

Describe alternatives you've considered Using a third party RAM Disk generator.

Additional context OVMF already provides a RAM Disk Driver, albeit I never tested it to see what it does and how it can be improved. Nor I have hear about it being used in bare metal contexts.

The usefulness of the RAM Disk heavily depends on whenever it can be possible to make it non volatile on reboots. For example, installing an OS will always force a reboot at some point, so a bare metal RAM Disk for installing OSes isn't really viable (I did tested a Linux-created RAM Disk in a host to use for a QEMU VM where I could bypass the reboot issue, and it was... interesing, from a performance standpoint). I know that certain Power States like S3 Sleep does not touch RAM contents and keeps it powered on so its contents remain intact, so a RAM Disk that survives warm reboots seems theorically possible. Also, rebooting so far seems inconsistent because I have seen many cases where a reboot is an automatic power cycle instead of a warm reboot, so it may be important to document what does NOT make the computer to lose RAM Disk data if relying heavily on this feature.

The performance of RAM Disks varies a lot: https://www.raymond.cc/blog/12-ram-disk-software-benchmarked-for-fastest-read-and-write-speed/ Given than Firmware should be able to provide the lowest-level possible setup for the RAM Disk (Since it can allocate the RAM for it then tell the OS than it has less RAM than physically installed), I would expect that it should be the most flexible one. I'm not sure if a virtual controller or whatever the OS thinks that manages the RAM Disk could be made to use Huge Pages or any other interesing performance trick.

Note than most of these tools can format the RAM Disk they generate thus from an user perspective they are ready to use, which may be needed to do from within Firmware since formatting the RAM Disk everytime it is created OS side before using can become a chore. I'm not intending than Firmware matches certain capabilities like automatic backup and restore of the RAM Disk contents on shut down and on, just creating it and not zeroing the allocated RAM on warm reboots is enough for basic functionality.

Also, a RAM Disk may be useful to do things like provide a virtual Drive with Drivers. For example, other propietary Firmwares relies on the WPBT (Windows Platform Binary Table) ACPI Table to conveniently provide Drivers for Windows to install (Which may be VERY useful for basic network connectivity, so NIC / WiFi Drivers), but its nature makes than the installer is executed in every boot, and it may automatically install things the user does not want. Creating a small dumb RAM Disk, uncompressing an internal ZIP on it, then let the user to manually point Windows to the Drivers is a less invasive way to achieve the same.

miczyg1 commented 1 year ago

We could include the RamDisk driver into UEFI Payload, but I would need some clear procedures how it i supposed to be used on some real examples.

I can see that HTTP boot uses the RamDisk driver for some virtual CD or virtual disk.

I know that certain Power States like S3 Sleep does not touch RAM contents and keeps it powered on so its contents remain intact, so a RAM Disk that survives warm reboots seems theorically possible... it may be important to document what does NOT make the computer to lose RAM Disk data if relying heavily on this feature

Anything that does not do power-cycle (so a simple reboot/reset) will not clear the memory and the RamDisk content will be present in the RAM. Intel TXT sometimes requests full memory clear. But one would need some additional software that will search for the RamDisk in its previous location (wherever it was allocated and placed in memory). So it is not just that simple. I can only see some specific and custom use cases here, so we can provide the necessary drivers to handle RamDisks from UEFI side, but the rest is up to the user.

zirblazer commented 1 year ago

but I would need some clear procedures how it i supposed to be used on some real examples.

The simplest use case will be just as an empty scratch disk that is visible from Windows to do CrystalDiskMark benchmarks and copy files to/from as if it was just another drive. At a basic level it doesn't need to do more than that.

For reference, I have used Passmark OSFMount and its functionality when creating an empty RAM Disk (You can also load an image file as a RAM Disk) is rather simple: It allows you to create a X sized RAM Disk (You type how much RAM you want, and it will reserve it) It can create a MBR or GPT Partition Table, or have no Partition Table at all It can format a single main partition as either FAT32, exFAT or NTFS (I suppose than if there is no Partition Table it will be treated as Superfloppy). Biggest omission is missing Sector Sizes when formatting. Multiple partitions may as well be equal to set them up as multiple RAM Disks as individual units... It can present it either as a fixed or removable device, read-write or read only It can emulate it either as a physical or logical device, HD, CD-ROM or Floppy It can assign it a Drive Letter so you can see it in Windows File Explorer. But, if Firmware creates the RAM Disk, I assume than whatever identifiers it uses could be exactly the same on every boot, so from Windows perpective I assume than it will most likely remember Drive Letter if the identifiers like path/location don't change.

What it can be used for depends on whenever it is volatile or not. I can see myself doing things like installing Windows 11 on a RAM Disk to measure install time without having any other disk drive plugged in. Equivalent to a volatile Windows LiveCD, without being a LiveCD

Intel TXT sometimes requests full memory clear.

In Z690 you don't have to worry about that :P

miczyg1 commented 1 year ago

Will add experimental (no documentation/usage guides whatsoever yet) RamDisk support to Z690 v1.1.3 and Z790 v0.9.1: https://github.com/Dasharo/edk2/pull/103

zirblazer commented 11 months ago

Tested the experimental RAM Disk driver. So far, these are the findings:

Dasharo does NOT save anything related to the RAM Disk configuration menu. Creating an empty RAM Disk or loading one from a file will work for the current session, but saving settings, rebooting and entering the Firmware setup again will show everything related to the RAM Disk as default (None created). This means that you have to configure the settings again on every boot.

The RAM Disk seems to be usable only under the UEFI Shell. While that is rather limited, it does actually have some uses as I have seen some people download stuff to a RAM Disk from the UEFI Shell then boot stuff from there, albeit I have no actual use case to test if doing that currently works.

Creating an empty RAM Disk shows a visible entry when using the map command on UEFI Shell, like this: map BLK14: Alias(s) : VirtualDisk(0x42C88018,0x42D88017,0) The size range (In hex) is actually displayed when you create it on the Firmware menu. The RAM Disk in this state is completely unusable. In theory, you can make it usable by formatting it, but currently I can't find tools to do that. Intel provides three tools as part of the UEFI Shell Disk Utilities but they are quite old. diskpart.efi still works, it shows me a Shell of some kind and using LIST command does list my 3 attached SSDs, which is not what I want. efichk.efi and efifmt.efi (Format) do absolutely nothing when invoked, and typing efichk or efifmt -h says that it doesn't have any help attached. Based on this, they used to work at some point a decade ago.

The other alternative was loading a preformatted file to use as disk storage, which I did: dd if=/dev/zero of=file.img bs=1 count=0 seek=128M mkfs.fat -F 32 file.img Putting file.img in a Firmware accessible location (Literally ANYTHING, cause it can read standard FAT32 but also EXT4 and NTFS. I put it on an USB Flash Drive) and loading it instead causes UEFI Shell to recognize it and automatically map it as a File System. Note that while you are browsing for files in the RAM Disk configuration menu, pressing Esc throws a File read error for no reason. Entering the File System in the UEFI Shell and typing vol gives me back expected size: fs9: vol 132135936 bytes total disk space 132135936 bytes available on disk

I didn't tried copying things to it from the UEFI Shell, but I would assume than any changes made to it would be volatile and file.img isn't actually modified, just loaded as the RAM Disk template.

Sadly, this is as far as I could get with the RAM Disk. My intention was that you could create and format a RAM Disk in Firmware then pass it as a sort of fixed and/or removable drive, ready to use, to an OS like Windows that would assign it a standard Drive Letter, so that you could point Chrome and Firefox caches to it, which is one popular usage. Doing this seems to requires a lot of extra work which may involve ACPI Tables to pass knowledge of the RAM Disk to an OS, as both Linux and Windows seems to be completely unaware of it. So far, there is another guy that is having fun with RAM Disk that pretty much did most of the research that would be needed to get this into an actually usable state: https://forum.level1techs.com/t/build-dev-blog-christmas-workstation-build-efi-ram-disk-fun/189871 https://forum.level1techs.com/t/devembary-2022-universal-ram-disk-bootstrapper/191607 Seems like a lot of work.

desowin commented 8 months ago

My use case is to use RAM Disk for temporary build files. Around 8 GiB is enough for me. The only functionality I would need is the auto format. The data is volatile and I do not care about it after reboot. I am currently using RAM Disk software for this, because the Dasharo RAM Disk does not show up in Windows (because as mentioned above, it would require modelling it as actual device).