Scan BIOS directories / structure... just like thumbnails.

[lasers]

After noticing that some cores requires lot of BIOS image files under the system directory, I was looking for a sensible way to keep my bios organized... as I do not want to throw every bios I can find in RetroArch directory.... or a single "system" directory... because it is easy to forget which file belongs to which core. Can RetroArch support something of a structure just like thumbnails... for bios? Few possibilities.

1) Could RetroArch generate empty bios directories on the fly? It would be nice, organized, and super easy for users to know where to put their bios in... although the directories are likely to be empty most of the time. It would be organized very nicely.

2) Could RetroArch support recursively scan the bios / "system" directory too? And people can set up directories up the way they like if they feels like it... or to throw everything in one place. It would be able to find the necessary files.

Here's what I'm going for...

.
└── .config/
    └── retroarch/
        ├── assets/
        ├── autoconfig/
        ├── bios/
        │   ├── Nintendo - Nintendo Entertainment System/
        │   │   ├── DISKSYS.ROM
        │   │   └── NstDatabase.xml
        │   ├── Nintendo - Super Nintendo Entertainment System/
        │   │   ├── cx4.data.rom
        │   │   ├── cx4.rom
        │   │   ├── dsp1.data.rom
        │   │   ├── dsp1.program.rom
        │   │   ├── dsp1.rom
        │   │   ├── dsp1b.data.rom
        │   │   ├── dsp1b.program.rom
        │   │   ├── dsp1b.rom
        │   │   ├── dsp2.data.rom
        │   │   ├── dsp2.program.rom
        │   │   ├── dsp2.rom
        │   │   ├── dsp3.data.rom
        │   │   ├── dsp3.program.rom
        │   │   ├── dsp3.rom
        │   │   ├── dsp4.data.rom
        │   │   ├── dsp4.program.rom
        │   │   ├── dsp4.rom
        │   │   ├── sgb.boot.rom
        │   │   ├── st010.data.rom
        │   │   ├── st010.program.rom
        │   │   ├── st010.rom
        │   │   ├── st011.data.rom
        │   │   ├── st011.program.rom
        │   │   ├── st011.rom
        │   │   ├── st018.data.rom
        │   │   ├── st018.program.rom
        │   │   └── st018.rom
        │   ├── Sega - Mega Drive - Genesis/
        │   └── Sony - Playstation/
        │       ├── scph5500.bin
        │       ├── scph5501.bin
        │       └── scph5502.bin
        ├── cheats/
        ├── config/
        ├── content_history.lpl
        ├── content_image_history.lpl
        ├── content_music_history.lpl
        ├── content_video_history.lpl
        ├── cores/
        ├── database/
        ├── downloads/
        ├── overlay/
        ├── playlists/
        ├── retroarch
        ├── retroarch-core-options.cfg
        ├── retroarch.cfg
        ├── screenshots/
        ├── shaders/
        └── thumbnails/

[kivutar]

I don't think it's worth the effort

[RobLoach]

1. Could RetroArch generate empty bios directories on the fly? It would be nice, organized, and super easy for users to know where to put their bios in... although the directories are likely to be empty most of the time. It would be organized very nicely.

There is libretro - Bios.dat that, when run through something like clrmamepro or RomCenter, will create a structure similar to what you proposed in the bios directory. This unfortunately doesn't really match up with what RetroArch expects for where the BIOS files should live.

We could merge the entries in libretro - bios.dat to have the files show up in one single folder to match what RetroArch expects.

1. Could RetroArch support recursively scan the bios / "system" directory too? And people can set up directories up the way they like if they feels like it... or to throw everything in one place. It would be able to find the necessary files.

The discovery of where those BIOS files live is mainly up to the Cores themselves. As @Kivutar mentioned, it would be quite a bit of work to go through each Core and adapt it to that new standard. There are some cases even, where the System/BIOS folder convention isn't considered. The whole process needs some work, so I'm glad you've started the discussion.

[Awakened0]

You can do this manually by using core overrides to set custom system directories for cores that need BIOS or other files.

[lasers]

@RobLoach: I tried mucking around with libretro - Bios.dat with clrmamepro. No such luck. I didn't understand how to use it and it does works fine, but a bit weird under linux. No worries though. Thank you.

The discovery of where those BIOS files live is mainly up to the Cores themselves

If necessary, could you make new directories? The standard could be just giving it its own "system-name" directory since most system cores seems to require same thing.

Looking at Reicast, it requires two files... in a directory?

~/.config/retroarch/dc/dc_boot.bin
~/.config/retroarch/dc/dc_boot.bin

could be...

~/.config/retroarch/bios/Sega - Dreamcast/dc/dc_boot.bin
~/.config/retroarch/bios/Sega - Dreamcast/dc/dc_flash.bin

I don't know. Just an idea. You do your thing.

@Awakened0: I tried what you suggested. It was not easy finding out what to name the override files and directories. I tried beetle_psx, mednafen_psx, mednafen_psx_hw, etc... I got it though.

retroarch/config
├── Mednafen PSX/
│   └── Mednafen PSX.cfg -> ../bios/Sony - Playstation
├── Mednafen PSX HW/
│   └── Mednafen PSX HW.cfg -> ../bios/Sony - Playstation
├── Nestopia/
│   └── Nestopia.cfg -> ../bios/Nintendo - Nintendo Entertainment System
├── bsnes/
│   └── bsnes.cfg  -> ../bios/Nintendo - Super Nintendo Entertainment System
└── remaps/

retoarch/bios
├── Nintendo - Nintendo Entertainment System/
│   ├── DISKSYS.ROM
│   └── NstDatabase.xml
├── Nintendo - Super Nintendo Entertainment System/
│   ├── cx4.data.rom
│   ├── cx4.rom
│   ├── dsp1.data.rom
│   ├── dsp1.program.rom
│   ├── dsp1.rom
│   ├── dsp1b.data.rom
│   ├── dsp1b.program.rom
│   ├── dsp1b.rom
│   ├── dsp2.data.rom
│   ├── dsp2.program.rom
│   ├── dsp2.rom
│   ├── dsp3.data.rom
│   ├── dsp3.program.rom
│   ├── dsp3.rom
│   ├── dsp4.data.rom
│   ├── dsp4.program.rom
│   ├── dsp4.rom
│   ├── sgb.boot.rom
│   ├── st010.data.rom
│   ├── st010.program.rom
│   ├── st010.rom
│   ├── st011.data.rom
│   ├── st011.program.rom
│   ├── st011.rom
│   ├── st018.data.rom
│   ├── st018.program.rom
│   └── st018.rom
├── Sega - Mega Drive - Genesis/
└── Sony - Playstation/
    ├── scph5500.bin
    ├── scph5501.bin
    └── scph5502.bin

It's not too bad. It's not bothersome as this...

retroarch/bios
├── DISKSYS.ROM
├── NstDatabase.xml
├── cx4.data.rom
├── cx4.rom
├── dsp1.data.rom
├── dsp1.program.rom
├── dsp1.rom
├── dsp1b.data.rom
├── dsp1b.program.rom
├── dsp1b.rom
├── dsp2.data.rom
├── dsp2.program.rom
├── dsp2.rom
├── dsp3.data.rom
├── dsp3.program.rom
├── dsp3.rom
├── dsp4.data.rom
├── dsp4.program.rom
├── dsp4.rom
├── scph5500.bin
├── scph5501.bin
├── scph5502.bin
├── sgb.boot.rom
├── st010.data.rom
├── st010.program.rom
├── st010.rom
├── st011.data.rom
├── st011.program.rom
├── st011.rom
├── st018.data.rom
├── st018.program.rom
└── st018.rom

This bios directory is messy with only 3 cores that depends on them and I can't tell which file belongs to which core. It's going to get more messy if I went ahead and set up all cores... and any future cores in the making that requires the files.

Thank you for that. The team may come up with something better in the future.

[RobLoach]

I didn't understand how to use it and it does works fine, but a bit weird under linux.

Will require Wine to get it to run under Linux. Just added a few fixes to the DAT to make it load correctly, thanks for bringing that up. But it does stand as a good review sheet to make sure you have all the files. Do you see anything that are missing from the list? https://github.com/libretro/libretro-database/blob/master/dat/libretro%20-%20Bios.dat

[lasers]

I didn't understand how to use it and it does works fine, but a bit weird under linux.

Day 1. My damn fault. I messed up an easy thing. I wget your link not knowing that it was a webpage. I couldn't load the damn thing.

Day 2. D'oh! I got it! With clrmamepro. We don't have one linux version? Ha.

I'm not familiar with DAT, CRC32, ROM Tools, etc... With help of this guide, https://github.com/retropie/retropie-setup/wiki/Managing-ROMs , I am able to build roms in Non-Merged Sets nicely (and zipped, I might add).

I'm not against using system_directory = "~/.config/retroarch/bios" or even default (unset) system directory if RetroArch were to be able to treat bios like games... -- Scan directories recursively for bios? -- Scan zipped archives for bios?

I thought this might work... (It didn't)

~/.config/retroarch/bios
├── Nintendo - Famicom Disk System.zip
├── Nintendo - Game Boy Advance.zip
├── Nintendo - Super Nintendo Entertainment System.zip
├── Sega - Dreamcast.zip
├── Sega - Mega Drive - Genesis.zip
├── Sega - Saturn.zip
└── Sony - Playstation.zip

I can't tell you if anything is missing from the list as I hadn't gotten to them all. I at least can mention that http://wiki.libretro.com/index.php?title=Main_Page need more work. I ran into about 10 empty pages, pages with no bios hashes or bios crc32, pages with little information. Not possible to compare between cores. I'm not complaining though.

I like clrmamepro and will use it in the future to rebuild more roms. Thanks!

[RobLoach]

https://github.com/retropie/retropie-setup/wiki/Managing-ROMs

That's a great article :+1:

D'oh! I got it! With clrmamepro. We don't have one linux version? Ha.

Unfortunately, no. Most of the tools out there are Windows-only. Luckily Wine gets us there.

I thought this might work... (It didn't)

Unfortunately not. RetroArch expects the files all to be extracted to the same directory. We may want to actually merge all the entries in the DAT to match that requirement. Then when you scan/fix with clrmamepro, it would rename them correctly in the same folder. Thoughts?

[lasers]

Didn't see your recent edit.

I'm comparing things. Using this as an example...

Nestopia requires the following BIOS image files for operation:
    NstDatabase.xml (NES Database)
    DISKSYS.ROM (Famicom Disk System BIOS)

http://wiki.libretro.com/index.php?title=Nestopia

You're missing DISKSYS.ROM for game ("Nintendo - Nintendo Entertainment System") in the database. It does exists in "Nintendo - Famicon Disk System" though. If actually merged, all entries in the DAT to match that requirement wouldn't be an issue. Is this an intended behavior or you're missing that in your database? I could try and check more. I think it might be former.

I think using clrmamepro (under linux, for one) or any other ROM manager could be a bit more work for users wanting to try and set up RetroArch / Lakka / portable toys / others. When I set up RetroArch, I did not know to check DAT / CRC32 / ROM Tools / etc... Just filename and hash. I assume most people just toss the files in and be done with it (as soon as the cores fires up without any issues.)

I also assume if people see empty bios directory in ~/.config/retroarch/bios along with empty "systemname" inside, they can put bios in there. It sounds like you're wanting to do same thing I initially did... which is to put all files in system_dir without any "systemname" dirs... only with DAT checks this time?

If that's the case, I still don't like it. If RetroArch do requires that, I could use core override (for all cores) instead as my preferred solution... only this time with your current DAT checks (and extracting the zips everytime I add something new).

Funny thing. For some reason, your database didn't grab my NstDatabase.xml... It was yet another bad github webpage wget. You saved me!

[andres-asm]

2) Could RetroArch support recursively scan the bios / "system" directory too? And people can set up directories up the way they like if they feels like it... or to throw everything in one place. It would be able to find the necessary files.

RetroArch doesn't read BIOS files for the core, the core itself needs to look for them, we just give them a "hint" about their location.

It could be doable I guess but not really worth the effort.

[andres-asm]

The only thing we could do is use core names like when using save state sorting. I think this is a won't fix.

I like my stuff tidy too but it won't scale without a lot of work in the core side of things and it would probably impose a naming scheme not everyone will agree with.

[lasers]

Your core Nestopia is dirty. I asked you about few days ago because I like my stuff tidy too. I'm expecting to see Nestopia UE, not Nestopia. https://github.com/libretro/RetroArch/issues/3864

[andres-asm]

Then send a pull request on either the core or the info file side. In any case this isn't a retroarch issue, it's a core issue.