Open Senemu opened 2 years ago
ghost
commented
2 years ago Yeah, I've noticed this before.
I've got some audio CDs with an absolute ton of index positions to mark different sections of the songs, they get lost on conversion, always have done even dating back to when CD support was first added.
cuavas
commented
2 years ago Index markers are just a subchannel field. If you use a raw sector format for the data, the index markers will be preserved correctly.
Senemu
commented
2 years ago Index markers are just a subchannel field. If you use a raw sector format for the data, the index markers will be preserved correctly.
This seems to work. Not in MAME, but I can extract the CHD to bin/toc and that works correctly in another Sega Saturn emulator.
bin/cue output is still incomplete, but chdman warns about that.
I have updated the title of this issue to clarify my original problem.
chdman discarding input data without warning is unexpected.
The bin/cue file should work to create a good CHD, since it contains all the required information.
Otherwise, there should be a warning about the conversion being lossy.
cuavas
commented
2 years ago Yeah, we’re aware not all TOC and CUE directives are supported (IIRC TOC zero fill is another thing that isn’t handled).
vmicka72
commented
3 months ago Indexes informations are lost when creating the CHD. Index informations are stored in Q-subdata. So reading CHD with sbi/lsd file containing subchannel datas is not sufficient to have all informations?
rb6502
commented
3 months ago CHD-CD was not intended as a bulletproof archival format, and people trying to use it as one are fooling themselves. Use CHD to save disk space, but always keep your original-format rips if you care about that stuff.
That said, index handling is significantly better after recent changes by @987123879113 . Multi-session discs, track flags, and indexes all have much better support now.
zakk4223
commented
4 days ago Indexes informations are lost when creating the CHD. Index informations are stored in Q-subdata. So reading CHD with sbi/lsd file containing subchannel datas is not sufficient to have all informations?
If you created a chd with sbi/lsd subchannel data AND the process preserved it in the CHD (which I believe it should). The indices are there, it's just that the applications consuming the CHD will have to extract the subchannel data themselves and parse through it for index data. If you created it from a cue file with INDEX elements, it will not do anything with those.
It doesn't feel like there's much support for extending the metadata tags for this stuff in the CHD, so my suggestion would be to only create your chds from rips that have the subchannel data, and encourage your application developers to consume the subchannel data appropriately.
HeroponRikiBestest
commented
4 days ago CHD-CD was not intended as a bulletproof archival format, and people trying to use it as one are fooling themselves. Use CHD to save disk space, but always keep your original-format rips if you care about that stuff.
That said, index handling is significantly better after recent changes by @987123879113 . Multi-session discs, track flags, and indexes all have much better support now.
While I definitely agree that CHD isn't a bulletproof archival format, and shouldn't be used for that purpose, the lack of proper index support does result in issues in games in mame already. There are likely at least a few games that are affected by this issue; drawing from redump's database, the list of games that have at least one track that goes up to index 02 or higher are:
One game I already know of that's impacted by this on MAME is Taito Chase H.Q. + S.C.I. on Sega Saturn. The issue is similar to the one described here: https://github.com/tpunix/SAROO/issues/136
The problem is that each game has only one audio track. They are multi-track [I think they mean multi-index here], contain several voices (radio) that must be played at different times and only play the first time.
In the fenrir it stays played on a loop
MAME suffers the same issue as well, using MAME's software list CHD for Saturn chase hq http://adb.arcadeitalia.net/?mess=chasehqs&list=saturn
sha256sum 798cd1eaa030757631518e0e766d3f5177221547a63277a81d545b50d07bd68e taito chase h.q. + s.c.i. (japan) [t-1105g].chd
https://github.com/user-attachments/assets/8701d08a-4826-43d0-a208-552e399e7177
The above is recorded from latest MAME commit. Each radio voice snippet is supposed to just play on the dashboard screen at the start of each level, but instead it just plays the entire track over and over for as long as you continue playing. Real disc on real hardware, for comparison:
https://youtu.be/pvJJkfn5uhQ?t=15
And, redump.org's cuesheet for the game, so you can see what I mean better:
http://redump.org/disc/23663/
(there are two games on this disc that use radio, hence two tracks having a bunch of indexes)
Taito Chase H.Q. + S.C.I. (Japan).cue.txt
While I haven't tested it, I saw the issue with the "Monster Slider" game that mentioned this issue 2 years ago at https://github.com/MiSTer-devel/Saturn_MiSTer/issues/123, and given what's described there, I'd assume that game likely has the same issues on MAME as it did in its linked MiSTer issue.
This is a particularly noticeable issue- SSF appears to have even had a specific hack for this game to make it work right- but given how many titles are in the text file I posted above, there's likely at least a few other games that will suffer from inaccuracies. Given that CHD is regardless the "main" format MAME uses for CD-based games, and the fact that redump and other similar databases for game dumping use cuesheets or similar approaches due to the inherent non-reproducibility of raw subchannel dumps, MAME should add support for storing cuesheet track indexes when converting to chd, or just generate the subchannel data using the track indexes from the cuesheet. Barring either of those, it should at least support reading track indexes from cuesheets a user supplies themselves. At best you could refine to 0 subchannel errors with two relatively pristine copies of the same disc and some luck, and even then, the limited error correction for subchannel data means there can still potentially be incorrect data.
If you created a chd with sbi/lsd subchannel data AND the process preserved it in the CHD (which I believe it should). The indices are there, it's just that the applications consuming the CHD will have to extract the subchannel data themselves and parse through it for index data.
Where did you hear this? SBI/LSD files are, to my knowledge, exclusively for storing the list of subchannel sectors where PSX libcrypt and pre-v4.75 PC SecuROM protected discs have deliberately incorrect CRC16 subchannel hashes, as well as the hashes for said subchannel sectors. I am not aware of any other data it stores beyond what is needed to either make libcrypt function, or make pre-v4.75 SecuROM discs function in the event that your ODE can't return the fallback mode flip check correctly, and that data certainly does not include track indexes.
zakk4223
commented
4 days ago Sorry, misspoke/was looking at an earlier comment and just autopiloted.
I meant .sub. I have some rips of CD+G audio discs in bin/cue/sub and digging in the SUB files it looks like they contain the Q-channel data (although I'm just staring at hex dumps so I could be wrong). CHD has a full 96 bytes/sector for subcode data so in theory it contains this too. I'll try to look at a converted CHD later to verify.
Unfortunately most of the systems that have this issue with multi indices seem to all be dumped with the index data in CUE files, not in a format that contains subchannel data.
I've already done a bit of work taking the changes from https://github.com/mamedev/mame/pull/12201 and getting chdman to insert those as additional metadata tags. However from the comments/discussion there it seems like mamedev would prefer the solution be actual subchannel data.
HeroponRikiBestest
commented
3 days ago Oh, .sub files sure, those contain the subchannel data for every subchannel (well, provided your dumping program dumps them properly, anyways). Either way, I hope mamedev will allow either index tags, or for chdman to generate subchannel data from the indexes, since for reproducibility/verifiability purposes, subchannel data for dumped discs generally isnt datted or saved (i.e. every redump dump made in the past ~decade has had the subchannel data dumped alongside the bin/cue, it's just not datted and ultimately discarded), and that won't change. (Except for cd+g like you mentioned, plus cd+eg, cd+midi, etc, since there's literally no other option, but as a result, any actual database with that will be a difficult challenge for obvious reasons.)
bikerspade
commented
3 days ago Index tags (and other tags) from the cuesheet are preferable vs. using a sub file. All the information in a cuesheet is deterministic (except for the filename of course) for a specific disc master, whereas .sub are never going to be deterministic in real-world practice.
p1pkin
commented
3 days ago @bikerspade
All the information in a cuesheet is deterministic (except for the filename of course) for a specific disc master, whereas .sub are never going to be deterministic in real-world practice.
this is not true. information in cue is EXACT SAME deterministic as subcodes read from CD disc, because all the information in cue is in fact information from subcodes, but obfuscated in its own cue-way.
Index tags (and other tags) from the cuesheet are preferable vs. using a sub file.
but why? just because Redump.org database and CUE format unable to handle proper original subcodes?
987123879113
commented
2 days ago Previously I discussed with cuavas about how to handle storing the additional info in CHDs and the answer was basically to generate subchannel data for dumps that don't include subchannel data, and to make the CD-ROM emulation properly handle the subchannel data. And for extracted a CHD to bin/cue, to parse the subchannel data and write the info.
My work in https://github.com/mamedev/mame/pull/12201 was simply to get the required data from the cues into MAME in some way because I needed to add multisession support to make some games work at all, and I brought some much requested friends along with it. All of the info was needed to be able to work on more of the CD-ROM implementation.
Data is identical, but audio track indexes beyond 01 are missing.