search

LTRData / ImDisk

ImDisk Virtual Disk Driver
GNU General Public License v2.0
426 stars 72 forks source link

Is this the right setup for RAM disk? #9

Closed T-640 closed 2 years ago

T-640 commented 2 years ago

I have decided to create RAM disk to relieve some pressure from SDD drive when compiling software to prevent wear out, but still something is writing a lot to the SSD during the compilation process with MSVC, despite the whole project being put in a RAM drive and compiled from there. The fact that it is done inside virtual machine is irrelevant, right?

I would like to make sure that I have correctly set up imDisk for this task. Could you take a look at the following screenshot and commands and tell that it indeed creates a partition in RAM and not anywhere else (like in the virtual memory of pagefile)?

Manually starting these drivers due to #8:

sc create imdisk type= kernel binpath= "C:\imdisk_test\imdisk.sys"
sc start imdisk
sc create awealloc type= kernel binpath= "C:\imdisk_test\awealloc.sys"
sc start awealloc

Then executing the cpl file:

Ramdisk

LTRData commented 2 years ago

Yes, this is correct. This creates a RAM disk i physical memory. The problem is however that Visual Studio uses the temp directory a lot for intermediate files during build, so you will still get lots of writes to SSD unless you also relocate the temp directory to a RAM disk.

T-640 commented 2 years ago

I see, thanks. Do you mind if I keep this issue open until I try it?

LTRData commented 2 years ago

No problem, keep it open and come back with your findings later!

T-640 commented 2 years ago

It does not seem to take any significant effect, still compiling the whole project from scratch wasted about 600 megabytes (reported by host OS).

Through sysdm.cpl I have changed TMP and TEMP environment variables from %USERPROFILE%\AppData\Local\Temp to R:\Temp, rebooted, started imDisk drivers, created a 1536 megabytes partition in physical memory, created the R:\Temp directory there, copied the project files and started compiling.

Is there any other place MSVC could write to? Can compilation process trigger writes to virtual memory?

I've got 8 gigabytes total on that computer, 6 for guest Windows 11 ARM (now 4,5 with RAM disk), and 2 for host OS (Macintosh OS). Both always at any time have at least 1 gigabyte of free RAM available (I do not run any other software neither on guest nor host besides MSVC), so it is not supposed to start using virtual memory (I hope so).

LTRData commented 2 years ago

I would suggest that you use Process Monitor to check what paths are written to and then try from there to analyze where those writes come from.

T-640 commented 2 years ago

Do you mean Process Monitor from Sysinternals? I have just tried it, excellent tool as it turns out, seems to be much better at showing what is going on with the disk than the built-in tools of Windows. It may take awhile, but I will share whatever I will have found.

LTRData commented 2 years ago

Yes exactly. It is the most widely used tool for these kinds of analysis on Windows systems. The tool is very actively developed and has been around for I think 20-25 years. It is really good.

T-640 commented 2 years ago

It appears that I have finally succeeded. Before I took all these measures, each day of work (around 8 - 10 hours) wasted more than 10 gigabytes, up to 20. Now it is no more than 4 gigabytes (9 hours 5 minutes of actively working one day - 3.78 gigabytes, and 11 hours 42 minutes of computer being mostly idle another day - 3.43 gigabytes). And that is for the whole system (guest OS + host OS).

ImDisk was instrumental in achieving this, much appreciated!

Such rate of writes is good, right? Although, perhaps I am overreacting? They say these SSDs can handle a lot of writes unlike USB sticks. Never had an SSD before (MacBook Air with M1 CPU), and dozens of HDDs that I had used have never failed me (some are as old as being back from the 90's, that apparently witnessed Windows 95, and whose manufacturers are long gone now). That is more than 20 years of exploitation for some of them, do SSDs serve as long, or at least 10 years?

Here is a list of additional things apart from setting up RAMdisk that I did to minimise the number of writes, as promised. A few notes before I begin:

1) My knowledge of Windows administration is limited, some may even say that I have no idea what I am doing, and perhaps they will be right;

2) The policy was "Disable first, ask questions later". I simply looked at whatever process was writing to disk during compilation, read some articles about what it does and if it seemed to be non-critical tried disabling it, without any particular plan.

3) These experiments were done inside virtual machine, so I had the luxury of being able to act boldly, something I would not dare trying on actual hardware.

4) There was no need for the Internet connection, so virtual machine was disconnected.

5) Notes may be messy and even incorrect at times.

6) I have omitted executable names and files they write to, because this post is already very long, let me know if you need them. I have also omitted particular steps done to disable certain Windows components, so let me know if you would like any details.

So, here they are:

1) First and foremost, as you suggested, TEMP and TMP environment variables must be changed to point to RAMDisk. Not only compiler is using these, but IDE as well (I am using Qt Creator), sometimes the 1 GB RAMDisk even ran out of free space;

2) Hibernation needs to be disabled. One time to my great surprise when I shut down the virtual machine the write count suddenly spiked at 1 gigabyte (the size of partition in RAM), so I booted the virtual machine again and witnessed the RAMDisk partition being restored! Disabled via powercfg /h off

3) Disable Microsoft Defender;

4) Disable CDPUserSvc service;

5) Disable Prefetch service;

6) Disable WinInet task;

7) Disable XtaCache. This is something I will probably reenable, as from what I read it is supposed to speed up execution of x86-x64 programs on ARM64 systems;

8) Disable Diagnostic Policy Service;

9) Disable Windows Event Log service;

10) Disable Windows Management Instrumentation service;

11) Disable Provisioning task;

12) Disable Capability Access Manager Service;

13) Disable Windows Modules Installer Worker. This is related to updates, but since virtual machine is disconnected from the Internet, I do not need it.

14) Disable SilentCleanup and StartComponentCleanup tasks (Disk Space Cleanup Manager for Windows);

15) Disable Microsoft Windows SMS Router Service;

16) Disable DefenderApiLogger and DefenderAuditLogger. Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Autologger\ seems to have various logging services referenced, so I suppose even more could be disabled from there.

17) There were some other services and tasks that I have disabled which are apparently not affected by compilation process, I could list them here if you wish.

I have disabled a lot of things, so I feared system might become unstable, but everything appears to work fine, at least for the single purpose of compiling software that I use this machine for, there are just a couple of small issues that do not affect my workflow.

And if I am not bothering too much, I would like to ask a few more questions:

1) Does the amount of RAM allocated to RAMDisk has to be an even number with a power of two? i. e., is it okay to allocate 1250 megabytes instead of 1024?

2) Since ImDisk can not be properly installed on ARM system, I have copied imdisk.sys and awealloc.sys (Are there any other files I need to copy?) and use this script to start drivers:

sc create imdisk type= kernel binpath= "C:\ImDisk\imdisk.sys"
sc start imdisk
sc create awealloc type= kernel binpath= "C:\ImDisk\awealloc.sys"
sc start awealloc

Then I am manually launching imdisk.cpl to create the RAMDisk and format it.

How do I automate this process so it does not have to be done manually each time ? In particular:

1) Create 1024 mb partition in physical memory; 2) Assign it letter R; 3) Format it to NTFS; 4) Create R:\Temp directory.

There is no need to save partition on shutdown, I simply plug in external HDD at the end of the work day and manually copy my progress there.

LTRData commented 2 years ago

First of all, I have laptops with Intel and Samsung SSD that are about 10 years old that I have not done anything with in particular to save them from excessive writes and they still work without any problems. Nowadays the problems should be even smaller with this. Some 15 years ago though there were frequently problems with SSD drives that failed after a few years of writing, I saw that often in embedded systems for example.

Re: installing on ARM: http://reboot.pro/index.php?showtopic=22236&p=219209

Re: setting up for auto-creating a RAM disk at system startup, have a look at the FAQ: http://reboot.pro/index.php?s=22117c45263bdb65dd4f02cf26e7c69c&showtopic=15593

T-640 commented 2 years ago

Excellent, thanks a lot!

Gitoffthelawn commented 2 years ago

@T-640 What tool(s) were you using to measure the quantity of bytes read/written to your drives?

T-640 commented 2 years ago

@Gitoffthelawn For individual processes - Process Monitor from within the guest OS. For the total bytes written during the whole session - Activity Monitor from host OS (macOS).

Gitoffthelawn commented 2 years ago

@T-640 Thanks! Do you happen to know if there is something like macOS's Activity Monitor, but for Windows?

BTW, Process Monitor no longer works on some versions of Windows because it's developed by a Microsoft employee, who in my perception, toes the corporate line and wants to push people to buy the latest version of Microsoft products, regardless of the problematic consequences.

LTRData commented 2 years ago

BTW, Process Monitor no longer works on some versions of Windows because it's developed by a Microsoft employee, who in my perception, toes the corporate line and wants to push people to buy the latest version of Microsoft products, regardless of the problematic consequences.

You mean that Process Monitor no longer works on older versions of Windows? How old versions?

Gitoffthelawn commented 2 years ago

BTW, Process Monitor no longer works on some versions of Windows because it's developed by a Microsoft employee, who in my perception, toes the corporate line and wants to push people to buy the latest version of Microsoft products, regardless of the problematic consequences.

You mean that Process Monitor no longer works on older versions of Windows? How old versions?

Yes. I can confirm that it no longer even works on Windows 7. In a few years, I imagine it will no longer even work with Windows 10, as MS is pushing to move people onto Windows 11, and will likely push people harder and harder (using their typical FUD tactics - Fear, Uncertainty & Doubt ) to get people off of Windows 10.

Ironically, the more they do this sort of nonsense, the more people switch to macOS or Linux.

LTRData commented 2 years ago

BTW, Process Monitor no longer works on some versions of Windows because it's developed by a Microsoft employee, who in my perception, toes the corporate line and wants to push people to buy the latest version of Microsoft products, regardless of the problematic consequences.

You mean that Process Monitor no longer works on older versions of Windows? How old versions?

Yes. I can confirm that it no longer even works on Windows 7. In a few years, I imagine it will no longer even work with Windows 10, as MS is pushing to move people onto Windows 11, and will likely push people harder and harder (using their typical FUD tactics - Fear, Uncertainty & Doubt ) to get people off of Windows 10.

Ironically, the more they do this sort of nonsense, the more people switch to macOS or Linux.

Ah, it is not surprising that no longer supported Windows versions do not work with latest versions of tools like that. It is a tool that use lots of internal "hacks" to retrieve various things and applications like that quickly become a mess if you try to support lots of different OS versions.

But yes, as soon as Windows 10 is no longer supported in about four years I would expect less of latest versions of tools like that to work on Windows 10 as well. I would recommend that if you really want to run old and unsupported versions of Windows (or other OS) you really should keep old versions of tools and applications you need that are verified to work on that OS version because quite naturally, OS vendors quickly drop support for tools for unsupported versions.

Gitoffthelawn commented 2 years ago

I wrote a nice & enjoyable (in my opinion!) response, but given that GitHub is now owned by MS, I'm not sure I want to post it here. ;)

A few brief points from what I wrote:

T-640 commented 2 years ago

@Gitoffthelawn

Do you happen to know if there is something like macOS's Activity Monitor, but for Windows?

I am afraid not, but before I discovered Activity Monitor I simply had used S.M.A.R.T. monitoring software to look at "Data Units Written" parameter right after booting and before shutting down. I used smartctl in particular.

Process Monitor no longer works on some versions of Windows

In case you have lost the version you need you can always download it from the Internet Archive's Wayback Machine

On a side note, I am happy to see that ImDisk still supports Windows XP!

LTRData commented 2 years ago

On a side note, I am happy to see that ImDisk still supports Windows XP!

ImDisk supports Windows versions all the way back to Windows NT 3.51 (with some features missing). It has not been actively developed for some years now and will be kept the way it currently is to provide an alternative for people who for various reasons use older Windows versions. We have Arsenal Image Mounter as an alternative for modern Windows versions and that is actively developed and new features are added there instead.

Gitoffthelawn commented 2 years ago

I am afraid not, but before I discovered Activity Monitor I simply had used S.M.A.R.T. monitoring software to look at "Data Units Written" parameter right after booting and before shutting down. I used smartctl in particular.

That's a great idea, and simple to automate. Thank you!

Process Monitor no longer works on some versions of Windows

In case you have lost the version you need you can always download it from [the Internet Archive's Wayback Machine]...

Another solid idea! I'm always saving things to the Wayback Machine so myself and others will be able to access the content in the future. There are some good browser extensions that make the process even faster.

On a side note, I am happy to see that ImDisk still supports Windows XP!

It's great that it still supports it. I bet RAM is cheap for old XP boxes, so people may be able to find uses for those old machines, especially if they are invigorated by the speed of a RAM drive. As long as XP boxes are used offline and not connected to a network or the internet, they should still work well.

Gitoffthelawn commented 2 years ago

ImDisk supports Windows versions all the way back to Windows NT 3.51 (with some features missing). It has not been actively developed for some years now and will be kept the way it currently is to provide an alternative for people who for various reasons use older Windows versions. We have Arsenal Image Mounter as an alternative for modern Windows versions and that is actively developed and new features are added there instead.

I spoke with someone recently, who by chance, uses Arsenal Image Mounter. They are quite pleased with it!

LTRData commented 2 years ago

ImDisk supports Windows versions all the way back to Windows NT 3.51 (with some features missing). It has not been actively developed for some years now and will be kept the way it currently is to provide an alternative for people who for various reasons use older Windows versions. We have Arsenal Image Mounter as an alternative for modern Windows versions and that is actively developed and new features are added there instead.

I spoke with someone recently, who by chance, uses Arsenal Image Mounter. They are quite pleased with it!

Glad to hear that! I use it a lot myself. I have almost not used ImDisk at all for several years now and since I am not actively working on it, I start to forget more and more how it was designed. I also recommend everyone who can to switch to Arsenal Image Mounter instead.

LTRData commented 2 years ago

On a side note, I am happy to see that ImDisk still supports Windows XP!

It's great that it still supports it. I bet RAM is cheap for old XP boxes, so people may be able to find uses for those old machines, especially if they are invigorated by the speed of a RAM drive. As long as XP boxes are used offline and not connected to a network or the internet, they should still work well.

ImDisk is also frequently used on XP to mount iso images as virtual DVD since there is no built-in support for that on XP.