Differencing VHD for OS installation

[andreyfedoseev]

Please consider adding support for differencing virtual hard disks for OS installation.

1. Base or Parent Disk: This is the main VHD file that contains the original data. This is where the OS is installed.

2. Differencing Disk: It only stores changes made to the parent disk. When a read or write operation is performed on the differencing disk, it refers back to the parent disk, and if necessary, layers its own changes on top. This is where the installed games would live (one child disk per game)

[andreyfedoseev]

PCEM seems to be using this library https://github.com/shermp/MiniVHD for working with VHDs. It claims to support differencing VHDs.

[schellingb]

Hello

Have you used the [ Boot and Install New Operating System ] feature already present in current DOSBox Pure? When running with the [ Run Installed Operating System ] it gives you a per-game D: drive which has its modification stored in a differencing disk saved as the 'save game' of that loaded content. So if you use RetroArch as your frontend, you will find the per-game differencing disk files in RetroArch's save directory named "-.sav".

Is this existing feature not covering the way you would like to use DOSBox Pure?

Currently uncompressed VHD files are already supported. But changes will be written back to the VHD file. How would you think the user interface would be to telling the emulator that you want changes written into a differencing disk file? Would that be a core option? Or would you somehow mark the VHD file as "wants differencing disk" (maybe with a specially file naming scheme)?

[andreyfedoseev]

Have you used the [ Boot and Install New Operating System ] feature already present in current DOSBox Pure? When running with the [ Run Installed Operating System ] it gives you a per-game D: drive which has its modification stored in a differencing disk saved as the 'save game' of that loaded content. So if you use RetroArch as your frontend, you will find the per-game differencing disk files in RetroArch's save directory named "-.sav".

Yes, I tried that. This is close to what I need, but not quite it.

I can install games on D:, but the installer still makes modifications to C: (start menu options, desktop shortcuts, etc.). I would like C: to become immutable after the OS is installed, so that different games don't collide with each other. I tried the option that prevents saving changes for C:, but after restarting it broke the first game it tried, since it relied on some files being present on C:.

How would you think the user interface would be to telling the emulator that you want changes written into a differencing disk file? Would that be a core option?

Yes, I was thinking of an option similar to Discard Disk Modifications. When enabled, it keeps the existing VHD intact and all changes are written to differencing VHD (one per game).

Of course, that would mean that the parent VHD is not allowed for changes after that, otherwise it would break the child VHDs.

[Torinde]

Dynamic and Differencing VHD support was recently added to DOSbox-X (joncampbell123/dosbox-x/pull/4273 and others) - in case re-using code from another DOSbox is more convenient.

[schellingb]

How about if we changed the option Advanced > Discard OS Disk Modifications (restart required) (with it's two values Off and On) to a new option Advanced > OS Disk Modifications (restart required) with 3 values Keep (Default), Discard and Keep Differences Per Content? An easier to understand title for that new third value would be even better.

But yes, if using such a new third value, then going back to Keep and then going to the Per Content again would cause horrible issues with games that were already used. Not sure we have a good way to prevent this. But maybe it's enough to mention in the core option description that after switching to the third value there's no going back.

[andreyfedoseev]

How about if we changed the option Advanced > Discard OS Disk Modifications (restart required) (with it's two values Off and On) to a new option Advanced > OS Disk Modifications (restart required) with 3 values Keep (Default), Discard and Keep Differences Per Content? An easier to understand title for that new third value would be even better.

That sounds great.

But yes, if using such a new third value, then going back to Keep and then going to the Per Content again would cause horrible issues with games that were already used. Not sure we have a good way to prevent this. But maybe it's enough to mention in the core option description that after switching to the third value there's no going back.

I'm personally not very concerned about that, given that such behavior will be expected and documented.

However, if you could somehow keep track of the parent/child relations, and also track the changes to the parent image, then you could at least notify the user that the child VHD cannot be used anymore due the changes to the parent.

[andreyfedoseev]

@schellingb

I just tested this feature using the latest Git revision, and it works great. Great job!

PS. I really like the new UI.

[schellingb]

Great that you could check it out from source! And even better that it's working great :-) I'm still working on some stuff for the next (big) release, so no official binaries just yet. Also really happy to hear you like the new UI! Was a bit shocked the other day when someone posted a side remark that made it sound like the new UI was a bug 😂

[andreyfedoseev]

After more thorough testing I think there's a problem with this feature.

What works:

1. I can select the new option in System settings
2. The changes I made to C: drive are saved to *-CDRIVE.sav file (or it seems so)

What doesn't work:

It seems that the file system gets corrupted somehow, and the amount of brokenness seems to depend on how many changes were made to C: drive.

For example, I do a clean install of Win 98, shut it down, enable the new option in System setting and launch the OS.

Then, I make a really small change, like creating a file on the desktop with a few words in it, and reboot (shut down the OS and re-launch). While starting, Windows complains about broken registry or broken system files, and offers to recover it. This works, but the content of my text file gets corrupted.

If a make a larger changes, like install a game from CD, this breaks the system completely. I get BSOD upon booting the OS.

Probably unrelated: while testing this I found another issue (let me know if you want me to file it separately). With the latest Git version I can't install Windows - the system hangs after I select:

1. Boot from CDROM
2. Start Windows Setup with CDROM support (or, whatever that text is)

It looks like it hangs while initializing the CDROM.

[schellingb]

@andreyfedoseev Thank you for testing this much more than I did 😅 I now fixed both issues you mentioned and tested it a bit more. I ended up defragging the hard drive which of course makes a huge differencing disk and it all worked out without Windows 98 complaining on startup.

[andreyfedoseev]

@schellingb Thank you for the quick fix! The issue with corrupted filesystem is resolved now, as well as the issue with CDROM initialization.

I noticed two things in the latest git revision:

1. Windows now detects a network adapter while being installed. I have to disable this device after the installation, otherwise it noticeably slows down the boot process.
2. Carmageddon 2 stopped working for some reason (it installs, but won't start with no errors). I remember testing it, but I probably did it with the latest official release of dosbox-pure.

Both issues are unrelated to the current one (#423), just FYI.

[schellingb]

Hey there

1. I looked into the slow boot and realized it's because TCP/IP with automatically obtained IP address makes Win9x wait for quite a long time on boot. I now added some code that intercepts the DHCP request packet and has the emulator send the OS a DHCP response with a generated IP address. It still takes about 2 seconds but I think it's good enough not to put the network adapter behind a core option. The network adapter exists for new multiplayer features introduced in 91f3504 which is quite neat when set up :-)

2. I tried a demo version of Carmageddon 2 and it worked for me in both latest Git revision as well as the last release 0.9.7. I then tried another version and it indeed did not work, it just froze the OS after the company logo intro movie, with both versions of the core. Can you try your version again with the released 0.9.7 and see if it actually worked? I'm not really sure why it doesn't work but Win 9x games in DOSBox are hit-and-miss... but we certainly don't want games to break in the next version update.

Thanks for all the testing and reports, I certainly don't mind it all being here, don't worry :-)

[schellingb]

From @i30817

Mmm. That's what assumed would be the default. I dump the whole OS into memory just so I don't have to copy on write both it and the game, so the os always starts from the same base state.

Doesn't appear to cause problems (it actually solves them because scandisk never happens to the os 'drive') because the os will get its time from the emulator at boot, so no savegames created later appearing earlier or anything. I can't wait for winebox personally, just to get rid of this scandisk business even in the game drive.

Good to hear you managed a differentiating os disk (alongside the game I presume). How did you do it, are you creating a new hdd image combining the stacks? Kept in memory or file? And how are the modifications stored, the whole hd file or something like xdelta? Or the actual modified files by comparing inside the hdd to the base one after dosbox exits?

Id really enjoy in file accessed markers and 10kb savegames didn't force me to waste several gb in a fvm games, so I'm quite interested.

Hard disk access happens in 512 bytes sized sectors. Differencing disk (and discard changes kept in-memory) store sectors that have been modified. This normally works out very pretty well with regular adding and removing files. Adding a new file will mark the sectors of the file system as modified and every sector that contains file contents. Removing a file will mark the file system only. Marked sectors are kept in the differencing disk. If a sector returns to its "original" 512 byte data it will get removed from the differencing disk. Now unlike DOS programs running in DOSBox, a booted OS has full raw access to the hard disk sectors so it is free to do what it wants. It can defrag the hard disk, or reformat and repartition the disk with a different file system. That's why we only handle raw sectors and don't try to figure out where actual files are, it might not be FAT16/32, it might be NTFS or EXT3 or someone's hobby OS running a custom filesystem.

[i30817]

Does that avoid the next boot scandisk because of forcefully shutting down windows with Ctrl f4 (even if nothing was being written)?

[schellingb]

With OS Disk Modifications = 'Discard' it does, where any changes to C: are discarded on shutdown. With the new OS Disk Modifications = 'Save Difference Per Content' forcefully shutting down the emulated OS will bring up Scandisk on the next boot. The way early Windows brings up Scandisk is super simple, it has a flag at the top of the file system which is set to 1 when booting up and 0 when shutting down. Because we're just writing whatever the OS writes, that flag is also saved. So when not properly shutting down the emulated OS it will see the flag being still 1 on the next boot and show Scandisk.

[i30817]

My experience with other dosbox and using separate hard drive images for the OS and game is that flag is per hard drive, so Ctrl f4 with a unmodified os hard drive image I didn't get it for the os drive but did get it for the dosgame.

I don't know of a way to avoid that if I want to use savegames and not savestates, or change a config for the game etc (I do think saving sectors is a major upgrade though).

Since you previously said you merge the game and the OS into a single runtime emulated c drive, I can't tell how you'd prevent this either...

I'm also curious how and if you are combining hard drive images (separate windows 95 and windows 95 game). Or is the game in bare zipped files? That would be useful, never to have to format a drive in windows 95 again...

[schellingb]

You still seem to be confused how two separate unrelated features of DOSBox Pure function.

1. This topic is about bootable, installed operating systems (usually Windows 95 or Windows 98). This process is initiated with the [ Boot and Install New Operating System ] option in the startmenu. Once installed, the OS is booted with [ Run Installed Operating System ]. When booting such an OS, the core will have the OS install hard drive as C: and the loaded game content as drive D:. Once the OS is booted, DOSBox doesn't really have control over the file system anymore because the OS has its own file system drivers that read and write raw sectors directly via an emulated BIOS.

2. The previous topic 421 was about shared system shells (usually Windows 3). That requires a user to get an installed Windows 3 or earlier installation, ZIP it up and place it as a .DOSZ file in the frontend's System directory. Because these early Windows versions are regular DOS programs, DOSBox retains full control over the file system and can do stuff like presenting multiple ZIPs overlaid onto each other as one unified C: drive. Windows 3 doesn't interact with raw disk sectors, it merely uses the DOS file functions which are fully emulated in DOSBox.

So with regular DOS games or Windows 3 and earlier, we retain full control over the file system and can interact on a much higher level thus everything can be just one drive and changes can be individually written out into a save ZIP file.

Whereas with a booted OS like Windows 9x we need to store changes made to C: or D: drive in a differencing disk which contains changed raw sectors (not files, we don't know what a file is at this point).

Hope that clarifies things a bit.

[i30817]

So to be clear you dont combine drives in windows 95 games and can't avoid scandisk if a windows 95 game writes to the drive and the os improperly shutdowns. That was what I wanted to know.

I think I may enter a request for enhancement to make windows 95 games in zips and not hd images bootable (by copying the contents of the zip inside the hd drive). That would be great.

[schellingb]

I can't think of a way we could combine the C: drive (the OS disk image) and the D: drive (the game content drive). There are ways of avoiding scandisk. We could just clear the dirty flag on startup and Windows would probably happily boot normally. The user can force shutdown the core while the emulated OS is in the process of writing the file system. This can potentially leave the file system corrupted. Such a corruption can potentially be fixed by Scandisk. It would be very hard for a user to know that a forceful shutdown would leave the file system corrupted.

So I'd flip don't and can't to: "we can't combine drives in windows 95 games and don't avoid scandisk if a windows 95 game writes to the drive and the os improperly shutdowns"

Windows 95 games in ZIPs is already supported, there is no need for copying. The core emulates a hard drive with a FAT file system that dynamically maps to the files inside the ZIP. Have you ever used the features we are talking about?

[i30817]

I wonder if you are already saving modifications of file by writing sectors to a cache apart if you couldn't just mark the scandisk flag when starting a sector write and clear it when there hasn't been a write 1 second after the last write (assuming the part of the code doing that doesn't know when a multiple sector write ends because it's only handling individual sector writes, otherwise clear it end all the writes in a sequence end).

And no I havent use the core for windows 95 before. I need to rework my games to not use conf files and a external VFS I used in the normal dosbox before trying (and extracting the games from their hdd images it seems).

[schellingb]

@andreyfedoseev I'm closing this issue now because it has been implemented. If you have any issues with it or something related, we can re-open the issue or just make a new one. Thanks for the request.