Feature request: Converter for MiSTer as well

[flynnsbit]

Two things are required to get the EXODOS collection setup for use on the AO486 MiSTer core.

1. Remove the parent directory structure in the zips from the collection to where the gamefiles are at the root of the zip and not a directory or two deep. I've created a powershell script as an example of how I am re-zipping them here: https://github.com/flynnsbit/scripts/blob/master/exodos_conversion.ps1

2. Now that AO486 has support for CD iso/bin mounting, the CD portions of the games need to be separated from the zip of the games and placed in a corresponding folder with the game title as the name so that the user can easily find and access them from the MiSTer storage and not from inside the VHD of the OS.

Not sure if you are interested at all in helping but I thought I would throw it out there. Love the idea of this converter.

[Voljega]

Don't know much about MiSTeR but yes I'm interested !

Still in vacations until the middle of the next week, after that I will woek on the previous request / fixes and then move on your request, which is a bit more complicated than the rest.

Le sam. 19 sept. 2020 à 15:07, flynnsbit notifications@github.com a écrit :

Two things are required to get the EXODOS collection setup for use on the AO486 MiSTer core.

1.

Remove the parent directory structure in the zips from the collection to where the gamefiles are at the root of the zip and not a directory or two deep. I've created a powershell script as an example of how I am re-zipping them here: https://github.com/flynnsbit/scripts/blob/master/exodos_conversion.ps1 2.

Now that AO486 has support for CD iso/bin mounting, the CD portions of the games need to be separated from the zip of the games and placed in a corresponding folder with the game title as the name so that the user can easily find and access them from the MiSTer storage and not from inside the VHD of the OS.

Not sure if you are interested at all in helping but I thought I would throw it out there. Love the idea of this converter.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Voljega/ExoDOSConverter/issues/11, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKXF4O3I6QE66CUVKG73RDSGSNG7ANCNFSM4RS7QKSQ .

[Voljega]

Hello @flynnsbit now I might start working on this feature, can you expand a bit on the first point (I don't read Powershell too well sorry) ? What I understood :

• You have a game let's say 3 Skulls of the Toltecs (1996).zip
• You uncompress the folder, you then get the 3SKULLS folder
• You go inside the this 3SKULLS folder then recompress all the inner files and dirs to a 3 Skulls of the Toltecs (1996).zip thus having all the inner files and dirs at the root of the new 3 Skulls of the Toltecs (1996).zip

Is that correct ?

Second part :

• before the previous recompression, all cds files from inside the 3SKULLS folder are to be placed in a separate folder (root of it) named 3 Skulls of the Toltecs (1996)

Is that correct ?

[flynnsbit]

This is going to be a bit of a long winded email but I wanted to give you the full picture as we have some new features added to the MiSTer FPGA as of last night. MiSTer has a PC compatible 486 core that allows for native 486 with L1 and L2 caches and VGA/SVGA support natively. It acts just like a 486 machine but also has the ability to mount VHD files up to 128GB Fat32 for DOS access, Floppies, and now CD support (missing redbook audio today but supposedly coming). As of last night, I now have a dos app that can tell the fpga to mount images as well (just like in Dosbox) and I can also control the CPU speed and caches from Dos and can hot swap all of that. So that is the base of all of this work.

Goal: Maximum amount of DOS games including CD support inside of a native dos system (so without the ExoDOS Launchbox frontend). To accomplish that I have been using Total Dos launcher (open source) to get the full library imported into the TDL front end and accessible from native dos. The closest I have come to a perfect setup is ExoDOS. Keeping in mind maximum game count and compatibility without having to configure 7000 games manually. Examples on my youtube, here is the latest unlisted that shows the full 7000 collection accessible in dos with loading screens, about, etc: https://www.youtube.com/watch?v=65sfXlLKVT0&feature=youtu.be

First part is carving out the CD isos/bins/cues into their own directory (long file name) with just those CD files in them, removing them from the master games that go in the VHD so that the entire collection is more like 60GB instead of 500GB. CD ISOs are stored in a separate area to allow for mounting outside of the dos VHD. More on the why for this later or see the video I posted above.

So based on this setup, I've changed a couple of things that I think keep the exo collection more pure, but allow for native dos to still use it.

You had my original idea right and that is what I was doing to get the long file names in the zip, but when extracted it would be the root of 3SKULLS. I think there is a better way. I have been creating all of this with Powershell but I think your solution could be cleaner for the community.

1. Exodos still includes the old front end before Launchbox that was called Meagre. The metadata for every game that is in the collection is in D:\eXoDOS\eXoDOS\Games!dos\2Fast4U\ as an example. (you probably already know this). I want to create 3 batch scripts that I can inject into the root of the 3 Skulls of the Toltecs (1996).zip that control the launch, about, and setup of that game.

   For Launch.Bat: That batch script can be auto-generated off of the 8.3 folder name inside the "3 Skulls of the Toltecs (1996).zip, grab the 3SKULLS folder name out of that, then match to the !dos\"8.3 folder name". Grab everything in dosbox.conf under [autoexec] so I know how far the path goes down in the zip, and the exe that was used to run. Build the launch.bat and insert into the corresponding zip. If a CD mount is detected in the dosbox.conf (EXAMPLE: @imgmount d "1.cue" "2.cue" "3.cue" "4.cue" -t iso) add a line to the launch.bat that calls a different dos program for the FPGA :(imgset.exe cd11 /cdgames/3skulls/3skulls.iso). If imgmount for a floppy is detected in the dosbox.conf then do the same thing but use (imgset.exe fdd1 /floppy/floppy.img). A 3rd option for non-fpga systems like a real dos machine would be to call a jpg viewer to ask the user to actually insert the cd or floppy disk in the system and then press any key. You can see this was the path I was going to take in the video above before I had the FPGA dev create the imgset utility)

For Setup.bat: this one is more complicated and might need to be a script that just parses each directory in the zip for exe/bat and shows an option menu to select what should be included in Setup.bat and then set the same paths that we did in Launch.bat. I haven't thought through another way to get setup.bat to have the right path to the setup file as that could be setup.exe, setup.com, install.exe. etc.

For About.exe: just grab the About.txt from each game out of D:\eXoDOS\eXoDOS\Games!dos\GAME\Meagre\About\About.txt and inject in the root of the exodos zip file. I also have some code to take the about.txt, /Meagre/Title/Title.JPG and create a splash screen for that interstitial screen before the game starts (again, demo in the video).

And that is where I am at today. Very close to having a nice runnable collection from a single interface in Dos.

Great seller that has pre-configured setups:https://misteraddons.com/products/mister-bundles He also sells the memory on amazon and you can get the DE10-Nano from mouser so very fast shipping for both. :)

On Fri, Oct 2, 2020 at 11:31 AM Voljega notifications@github.com wrote:

Hello @flynnsbit https://github.com/flynnsbit now I might start working on this feature, can you expand a bit on the first point (I don't read Powershell too well sorry) ? What I understood :

• You have a game let's say 3 Skulls of the Toltecs (1996).zip
• You uncompress the folder, you then get the 3SKULLS folder
• You go inside the this 3SKULLS folder then recompress all the inner files and dirs to a 3 Skulls of the Toltecs (1996).zip thus having all the inner files and dirs at the root of the new 3 Skulls of the Toltecs (1996).zip

Is that correct ?

Second part :

• before the previous recompression, all cds files from inside the 3SKULLS folder are to be placed in a separate folder (root of it) named 3 Skulls of the Toltecs (1996)

Is that correct ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Voljega/ExoDOSConverter/issues/11#issuecomment-702832298, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHXHK6F2HM6MP532IC23BFDSIX55FANCNFSM4RS7QKSQ .

[Voljega]

OK this is a lot to digest :)

I read everything, got some grasp of it, but will have a look at the video before commenting again.

I think it's better we start with baby steps, on my next comment, I will describe what my converter does at the moment, and how we can write post conversion for MiSter based on your comment and your video

Any way old Meagre metadata or new Launchbox metadata (which I'm using now) it's quite the same, juste a mater of writing metadata somewhere, those are the same, the about.Txt of old meagre metadata can be found easily in the new Luanchbox metadata. It's better too because Meagre metadata will anyway totally disappear in ExoDOS V5

[Voljega]

OK so here's what I'm doing at the moment :

• get metadata for the game (all txt data, image, manual) and copy it to target dir
• extract 3SKULLS game folder from 3 Skulls of the Toltecs (1996).zip
• copy 3SKULLS folder to target dir in a 3SKULLS.pc dir
• get the dosbox.conf from exodos collection and copy it to 3SKULLS.pc dir
• divide dosbox.conf into dosbox.cfg, commenting the [autoexec] part in dosbox.cfg and copying the same [autoexec] part to dosbox.bat
• parse dosbox.bat and modify it and all the cd files it references to be fully compatible with dos name format

Here's the sample log of the extraction :

>>> 1/1 >>> 3SKULLS: starting conversion
  Metadata: 3 Skulls of the Toltecs (1996), genres: Adventure
  computed genre Adventure-Visual
  copy pics and manual
  needs unzipping...
  unzipping 3 Skulls of the Toltecs (1996).zip
  unzipped
  copy game data
  copy dosbox conf
  create dosbox.bat
    clean single imgmount
      renamed 3skulls.cue to 3skulls.cue
      renamed 3skulls.bin to 3skulls.bin
      convert cue content -> FILE "3skulls.bin" BINARY
    imgmount path: imgmount d .\Games\3SKULLS\cd\3skulls.cue -t cdrom --> imgmount d .\3SKULLS\cd\3skulls.cue -t cdrom
    cd command: 'cd 3skulls' -> path is game name and no existing subpath, removed

Here's what the final folder looks like :

(The txt file named like the game actually contains the about text)

After that, let's do the post conversion part for MiSTer. So first step, what do we do ?

[flynnsbit]

Step 1: some simple clean up, rename dosbox.bat to launch.bat when looking in that bat, will the relative path into the 3skulls directory still work, if you were to execute it from the 3skulls.pc directory? We still need the long filename for the zip name as the indexer from Total Dos Launcher will need that name so it can insert it into the menu, Total dos launcher will auto convert the zip once it indexes down to the 8.3 format for dos.

Step 2: Convert imgmount command using -iso mount to misters version which is in the following format:

imgset ide10 cdgames/gamefolder/game.iso

(or cue) This file can actually be downloaded here if you wanted to see it, you would need a mister for it to function though. https://github.com/MiSTer-devel/ao486_MiSTer/blob/master/releases/drv/imgset.exe

Step 3: convert imgmount games that are mounting floppy instead to use imgset fdd0 /floppy/filename.img Step 4: I can handle this, but I am auto creating an About.JPG file that combine the About.txt and the Title.jpg for a nicer About screen for the games that could also include quick tips on keyboard controls. Then a script that runs showJPG.exe to show it. Like this https://youtu.be/o0bMVJ-5ku4?t=20

I think that is it.

[Voljega]

OK Step 1 : Keeping the file structure I posted earlier, the inside of the SKULLS3.pc folder (where dosbox.bat) is considered as C: by most version of dosbox on linux (by the way do you need an explicit mount for C: ?) and as '.' for all relative path starting from here

So before doing the operations for MiSter, this is what I have inside :

imgmount d .\3SKULLS\cd\3skulls.cue -t cdrom
pause
c:
cd 3SKULLS
call run

(run is the dos executable of the game)

and when you say

We still need the long filename for the zip name as the indexer from Total Dos Launcher will need that name so it can insert it into the menu

I guess we also need a fifth step which is recompressing either 3SKULLS or the content of 3SKULLS into 3 Skulls of the Toltecs (1996).txt ?

I'm also okay to do Step 4, provided we find something to do it in Python as it must also work on Linux now

[flynnsbit]

yes you got it right on all account. We do not need the "c:" call out as this will be running from within the directory it was extracted to automatically so it's relative paths from there. Agreed on the 5th step, a re-zip with the long filename leaving your structure at the root. then zip that up exactly as it sits but with the long filename for the zip name.

On Sat, Oct 3, 2020 at 4:05 AM Voljega notifications@github.com wrote:

OK Step 1 : Keeping the file structure I posted earlier, the inside of the SKULLS3.pc folder (where dosbox.bat) is considered as C: by most version of dosbox on linux (by the way do you need an explicit mount for C: ?) and as '.' for all relative path starting from here

So before doing the operations for MiSter, this is what I have inside :

imgmount d .\3SKULLS\cd\3skulls.cue -t cdrom pause c: cd 3SKULLS call run

(run is the dos executable of the game)

and when you say

We still need the long filename for the zip name as the indexer from Total Dos Launcher will need that name so it can insert it into the menu

I guess we also need a fifth step which is recompressing either 3SKULLS or the content of 3SKULLS into 3 Skulls of the Toltecs (1996).txt ?

I'm also okay to do Step 4, provided we find something to do it in Python as it must also work on Linux now

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Voljega/ExoDOSConverter/issues/11#issuecomment-703072775, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHXHK6HBHWED57CVJWQQ35TSI3SM7ANCNFSM4RS7QKSQ .

[Voljega]

So I started implementing the first step, here's the result with 3 simpler games (no cd, no floppy mounting) : https://www25.zippyshare.com/v/139lZxdA/file.html

Is is what you want ? or do you want a flat list of these files a the same level, without the *.pc folder structure ? In that case, how to handle the multiple launch.bat with same name ?

[flynnsbit]

Close. The games can stay zipped up with the original name from ExoDOS- More like this: Doom1993.zip and then inside of that at the root of that zip: and then one level down inside of that zip inside the doom folder within is the actual game, just like how the exo library has it.

So to find the path for the actual exe that needs to be in launch.bat, that is where the bottom part of the[Autoexec] area of the dosbox.conf

And you don't need the .pc structure at all, it can be exactly how exo has it: in your example it would be three zip files named this way:

Then inside of each it would look like this:

And then for launch.bat the correct path and exe/bat to run for the game is pulled from the [autoexec] area. Solaunch.bat would look like this inside of it based on what it got out of dosbox.conf:

cd DukeN11
cd Duke1
@cls
@dn1
exit

Option 1: for Setup.bat: it might just have to be set to do the same but run setup.exe. Option 2: would be for the python script that is building all of this to just show what exe/bat are in the game folder and prompt the user that is "building" it to select the correct exe for setup. That way I could at least quickly get through the 7000 games and set the right exe you see for Setup/Install.exe etc.

The point of all of this is those batch scripts are required to be in the root of the zip file and the scripts have to know what directory to "CD" into to run the actual game.

[flynnsbit]

Video demo of how it looks once you get it over to the launcher in the MiSTer: https://www.youtube.com/watch?v=CoupeozOhqw&feature=youtu.be

[Voljega]

everything from the [autoexec] part is in the launch.bat with correct path I have actually no idea at all what should go in the setup.bat part ? AS al games from EXO are already installed and preconfigured ?

In the meantine here's the modified generation : https://www120.zippyshare.com/v/qdRv7hFY/file.html

[flynnsbit]

for setup.bat, can you just prompt the person running the python script to pick a choice and show what exe/com/bat files are in the list zip folder. And then either allow typing that or selecting it SETUP (1 install.exe (2 setup.exe (3 setup.com (4 none, do not create a setup.bat? That way I can at least make a logical choice when I am building out the primary image. Yes, I would have to do that 7000 items but it would give the users of the front end an easy way to get to the setup for sound, video, etc. Some of the exo stuff is set to VGA and we want to drop the video down sometimes, re-configure the sound to use Midi instead of SB. ETc.

[flynnsbit]

You can remove the C: from all of the launches. That isn't needed. Everything else looks PERFECT!!!! Let me know when you are ready and I can try and test it against my collection on the MiSTer

[Voljega]

The setup.bat thing seems quite impossible and overkill, I can hardly scan the whole game directory for all bat com and everything,as they can be named setup, install, or basically anything, and most of the time this will on the cd when there's one and not even in the game dir install itself.

And the configuration chosen by Exo should be the best one everytime I guess, no need to change it.

Nevertheless, any files you might call won't work at all outside the dosbox 'virtual machine' I guess ? And if it works it would modify the game dir again, thus killing the whole purpose of the zip ?

"That way I can at least make a logical choice when I am building out the primary image." -> don't understand this too much either do you mean the VHD thing ?

I can remove the C: but if it's just that's not really needed but harmless, I'd rather leave it than doing another costly operation for nothing

[flynnsbit]

I hear you on setup.bat. No big deal, just skip it.

All of these files are not running from C: on the MiSTer. so as soon as that line runs you are going to kill everything else in that script. These files can be on D: or E: or F: depending on the setup.. I can just remove c: on my side if it is that big of a deal for you.

[flynnsbit]

Video for why no C: https://www.youtube.com/watch?v=MTJ7az9LsvI&feature=youtu.be

flynnsbit#1986 if you ever want to chat on Discord...

[Voljega]

No problem, if it's mandatory I will do it :)

[Voljega]

Guess we can close this one and continue with enhancement issues if needed, as we have gotten pretty far