emmercm / igir

🕹 A zero-setup ROM collection manager that sorts, filters, extracts or archives, patches, and reports on collections of any size on any OS.
https://igir.io/
GNU General Public License v3.0
381 stars 18 forks source link

Erroneous `Cannot move different files to` warnings resulting in improper behavior #440

Closed chainsawsalad closed 1 year ago

chainsawsalad commented 1 year ago

Paste the command

npx -yes igir@latest -vvv move test report --dat ../DATs/Redump/NEC\ -\ PC\ Engine\ CD\ \&\ TurboGrafx\ CD\ -\ Datfile\ \(541\)\ \(2023-04-09\ 22-11-42\).dat --input ./input --output ./output

> ls -l ./input/
total 514436
-rwxrwxrwx  1 jeff users 368156053 May 20 13:28 'Fighting Street (USA).zip'
-rwxrwxrwx+ 1 jeff users 158625630 May 20 13:23 'Jack Nicklaus Turbo Golf (USA).zip'
> ls -l ./output/
total 0

Describe the bug

Certain optical media files seem to behave unexpectedly when processed against certain other optical media files. I haven't been able to figure out why this happens, but I've isolated many examples of it. It results in a failure to copy or move one or both files.

In the below example igir fails to identify Jack Nicklaus Turbo Golf (USA).zip as a candidate when it should, and it spits out the following warning:

[13:28:17.360] WARN:  CandidateFilter: Cannot move different files to: output/Jack Nicklaus Turbo Golf (USA).zip:
[13:28:17.360] WARN:  CandidateFilter:   input/Jack Nicklaus Turbo Golf (USA).zip
[13:28:17.360] WARN:  CandidateFilter:   input/Fighting Street (USA).zip
[13:28:17.366] TRACE: CandidateFilter: NEC - PC Engine CD & TurboGrafx CD: Jack Nicklaus Turbo Golf (USA): found 0 candidates

Note the conflict with Fighting Street (USA).zip. This file is later successfully and properly identified as a candidate and is moved or copied as such. If I remove Fighting Street (USA).zip from ./input/, Jack Nicklaus Turbo Golf (USA).zip is properly identified as a candidate and the tool behaves as expected. I cannot reproduce this behavior with any other combination of files against Jack Nicklaus Turbo Golf (USA).zip.

Here are is a very short, incomplete list of other conflicts I've encountered (just TurboGrafx, but I see this for many other optical-based system DATs as well):

Zero Wing (Japan).zip + Dungeon Master - Theron's Quest (Japan).zip
Gain Ground SX (Japan).zip + Alnam no Kiba - Juuzoku Juuni Shinto Densetsu (Japan) (FAAT).zip
Hihou Densetsu - Chris no Bouken (Japan).zip + A.III. - A Ressha de Ikou III (Japan).zip
Super Air Zonk - Rockabilly-Paradise (USA).zip + Bonk III - Bonk's Big Adventure (USA).zip + CD Denjin - Rockabilly Tengoku (Japan).zip
Shadow of the Beast (USA).zip + Bonk III - Bonk's Big Adventure (USA).zip + Shadow of the Beast - Mashou no Okite (Japan).zip
Loom (USA).zip + Loom (Japan).zip
....hundreds more combinations

More details:

Excerpt from NEC - PC Engine CD & TurboGrafx CD - Datfile (541) (2023-04-09 22-11-42).dat:

    <game name="Jack Nicklaus Turbo Golf (USA)">
        <category>Games</category>
        <description>Jack Nicklaus Turbo Golf (USA)</description>
        <rom name="Jack Nicklaus Turbo Golf (USA).cue" size="1224" crc="93f9b06c" md5="591f2f3dd07461dc990b089e0963a719" sha1="b2a200c9a87cb22080192dcc899f56a4f624db86"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 01).bin" size="7914480" crc="17623548" md5="77232c745e40cc155ad72c1b79780986" sha1="e1baaf01eee215def8bd6f6aa080e3a41ffc74f4"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 02).bin" size="22946112" crc="c9cdcb10" md5="b30c100096e78328c3ef80c06d794b53" sha1="5f59a0080a26f35c25ac8de25ebf22c83dc75406"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 03).bin" size="11005008" crc="7df2dc3e" md5="0a08889b2cff35740f982bfca0abd34c" sha1="dc58e79d38cb85bb54e3c15a9e03e3744cf24428"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 04).bin" size="27723024" crc="b4580daf" md5="1ce418ce541cdd10854ffd78b02fd7d5" sha1="c1d0ff11a7c0b63c031083a1e1b306676e96cfba"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 05).bin" size="5407248" crc="cdcf16e9" md5="5587af5d843eaf719e80fdae2f1264c7" sha1="c90a80e4be09ed5aac6c76c5b1e31434358649ef"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 06).bin" size="24576048" crc="cd070c6f" md5="36d015776a4ac44d9daf250b336d0dfb" sha1="ecaefc831654200e78775985244af5b004f3c762"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 07).bin" size="26789280" crc="af06ac59" md5="f90dfebef960f5358c295e8e929d2d99" sha1="54207863f1e5fb82815537fac682c15c38c76227"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 08).bin" size="26624640" crc="19309a5a" md5="c1d7b1ec9a78fbb177fadd1f6ec10b02" sha1="2675e4d635a94e77df277db183dce57d6ed75bfb"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 09).bin" size="18449088" crc="615709b5" md5="9e3dd07f45d13238e79efbadd2a1f4f3" sha1="3b3cfbe0dc1d797117713b7881991d72995f112d"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 10).bin" size="13945008" crc="07842367" md5="e1e698803b01feaad7e9f82857621a0d" sha1="b45baea264810cd80e439784fb010626a47ad7a7"/>
        <rom name="Jack Nicklaus Turbo Golf (USA) (Track 11).bin" size="22595664" crc="025adc5c" md5="bf5ea8a875a5e8a2547bf856c0de0721" sha1="789012ea63cbbfedac32cf013da4c2f8845cd574"/>
    </game>

Hashes on disk (they match the dat file):

> sha1sum Jack\ Nicklaus\ Turbo\ Golf\ \(USA\)*
e1baaf01eee215def8bd6f6aa080e3a41ffc74f4  Jack Nicklaus Turbo Golf (USA) (Track 01).bin
5f59a0080a26f35c25ac8de25ebf22c83dc75406  Jack Nicklaus Turbo Golf (USA) (Track 02).bin
dc58e79d38cb85bb54e3c15a9e03e3744cf24428  Jack Nicklaus Turbo Golf (USA) (Track 03).bin
c1d0ff11a7c0b63c031083a1e1b306676e96cfba  Jack Nicklaus Turbo Golf (USA) (Track 04).bin
c90a80e4be09ed5aac6c76c5b1e31434358649ef  Jack Nicklaus Turbo Golf (USA) (Track 05).bin
ecaefc831654200e78775985244af5b004f3c762  Jack Nicklaus Turbo Golf (USA) (Track 06).bin
54207863f1e5fb82815537fac682c15c38c76227  Jack Nicklaus Turbo Golf (USA) (Track 07).bin
2675e4d635a94e77df277db183dce57d6ed75bfb  Jack Nicklaus Turbo Golf (USA) (Track 08).bin
3b3cfbe0dc1d797117713b7881991d72995f112d  Jack Nicklaus Turbo Golf (USA) (Track 09).bin
b45baea264810cd80e439784fb010626a47ad7a7  Jack Nicklaus Turbo Golf (USA) (Track 10).bin
789012ea63cbbfedac32cf013da4c2f8845cd574  Jack Nicklaus Turbo Golf (USA) (Track 11).bin
b2a200c9a87cb22080192dcc899f56a4f624db86  Jack Nicklaus Turbo Golf (USA).cue

Expected behavior

  1. I expect otherwise positive matching candidates to be treated as such, and not conflict with anything else going on.

  2. The log message let message = `Cannot ${this.options.writeString()} different files to: ${duplicateOutput}:`; is unintuitive and I honestly don't understand what it means when this issue is encountered. Maybe clearer messaging would help?

  3. Unrelated to this behavior specifically, but the console is flooded with missing XX files for: <unrelated ROM name>. In my example of using only two ROMs in ./input/, I see dozens of warnings of what look like attempts to match one of the ROMs to the wrong candidate, and the tool being overly helpful in explaining why. For example:

    [13:46:44.561] WARN:  CandidateFilter: NEC - PC Engine CD & TurboGrafx CD: missing 43 files for: Ys Book I & II (USA)
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA).cue
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 02).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 03).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 04).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 05).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 06).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 07).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 08).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 09).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 10).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 11).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 12).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 13).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 14).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 15).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 16).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 17).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 18).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 19).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 20).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 21).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 22).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 23).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 24).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 25).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 26).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 27).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 28).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 29).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 30).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 31).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 32).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 33).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 34).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 35).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 36).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 37).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 38).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 39).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 40).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 41).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 42).bin
    [13:46:44.561] WARN:  CandidateFilter:   Ys Book I & II (USA) (Track 43).bin
    [13:46:44.576] TRACE: CandidateFilter: NEC - PC Engine CD & TurboGrafx CD: Ys Book I & II (USA): found 0 candidates

    This is true, but Ys Book I & II (USA) is not among the ROMs in ./input/. This doesn't feel like it rises to the occasion of needing a WARN, especially because it's not doing this for every ROM in the DAT, just a subset. See the attached output for a better overview.

Debug logs

log.txt

igir version

1.6.4

Node.js version

18.12.0

Operating system

Linux

Additional context

No response

emmercm commented 1 year ago

I'm sorry @chainsawsalad, I had read this and had been thinking about it, and then never got back to you.

This is because a number of PC Engine CD discs share the same track 1 it looks like, with the SHA1 e1baaf01eee215def8bd6f6aa080e3a41ffc74f4. Internally, igir de-duplicates input files, because the thought was it doesn't particularly matter where a file of a certain hash comes from. But in the case of moving games with multiple files (e.g. discs), we do kind of care, and would prefer the files come from the same archive.

This is going to take quite a bit of internal refactoring to tackle, to break that input file assumption. I'll see what I can do!

On the logging, I agree with you, that's really aggressive. I'll drop the level down to DEBUG.

emmercm commented 1 year ago

Hi @chainsawsalad, I've been hacking at this slowly over the past few weeks, and I think I might have the solution.

Could you help me by running the testing instructions at https://github.com/emmercm/igir/pull/467#issuecomment-1629761257 against your local setup? I spent some time being able to run igir from a GitHub branch/PR.

lostb1t commented 1 year ago

@emmercm just tried it as i ran in the same issue for some ps games. Works like a charm for those games.

chainsawsalad commented 1 year ago

@emmercm I re-ran my test example with the original two ROMs, and things look good, however I ran it against the rest of my library and I do still see examples of this happening with other ROMs.

Working example:

npm exec --yes -- "github:emmercm/igir#emmercm/same-input-file-for-everything" -vvv move test report --dat ../../DATs/Redump/NEC\ -\ PC\ Engine\ CD\ \&\ TurboGrafx\ CD\ -\ Datfile\ \(541\)\ \(2023-04-09\ 22-11-42\).dat --input ./input --output ./output
> ls -l ./output/
total 514436
-rwxrwxrwx  1 jeff users 368156053 May 20 13:28 'Fighting Street (USA).zip'
-rwxrwxrwx+ 1 jeff users 158625630 May 20 13:23 'Jack Nicklaus Turbo Golf (USA).zip'

successful_log.txt

Not working examples, extracted out of command output:

[21:20:12.938] WARN:  CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN:  CandidateFilter:   /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN:  CandidateFilter:   /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:27.716] WARN:  CandidateFilter: Cannot move different files to: /output/Bonanza Bros. (Japan).zip:
[21:21:27.716] WARN:  CandidateFilter:   /input/Bonanza Bros. (Japan).zip
[21:21:27.716] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:21:30.518] WARN:  CandidateFilter: Cannot move different files to: /output/Record of Lodoss War II (Japan) (Demo).zip:
[21:21:30.518] WARN:  CandidateFilter:   /input/Record of Lodoss War II (Japan) (Demo).zip
[21:21:30.518] WARN:  CandidateFilter:   /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:32.957] WARN:  CandidateFilter: Cannot move different files to: /output/Faceball (Japan) (Demo).zip:
[21:21:32.957] WARN:  CandidateFilter:   /input/Faceball (Japan) (Demo).zip
[21:21:32.957] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:22:13.799] WARN:  CandidateFilter: Cannot move different files to: /output/Travel Eple (Japan).zip:
[21:22:13.799] WARN:  CandidateFilter:   /input/Travel Eple (Japan).zip
[21:22:13.799] WARN:  CandidateFilter:   /input/Gradius II - Gofer no Yabou (Japan).zip
[21:23:54.717] WARN:  CandidateFilter: Cannot move different files to: /output/Martial Champion (Japan).zip:
[21:23:54.717] WARN:  CandidateFilter:   /input/Martial Champion (Japan).zip
[21:23:54.717] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:23:54.717] WARN:  CandidateFilter:   /input/Fighting Street (Japan) (Rev 5).zip
[21:24:37.618] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip:
[21:24:37.618] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip
[21:24:37.618] WARN:  CandidateFilter:   /input/CDI Karaoke 10 (Japan).zip
[21:24:37.618] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM (Japan).zip
[21:24:37.618] WARN:  CandidateFilter:   /input/Lords of Thunder (USA).zip
[21:24:37.618] WARN:  CandidateFilter:   /input/Chou Eiyuu Densetsu - Dynastic Hero (Japan).zip
[21:24:39.419] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip:
[21:24:39.419] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip
[21:24:39.419] WARN:  CandidateFilter:   /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:39.419] WARN:  CandidateFilter:   /input/CDI Karaoke 10 (Japan).zip
[21:24:39.419] WARN:  CandidateFilter:   /input/Might and Magic III - Isles of Terra (Japan).zip
[21:24:39.419] WARN:  CandidateFilter:   /input/Blood Gear (Japan).zip
[21:24:46.020] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip:
[21:24:46.020] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip
[21:24:46.020] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.020] WARN:  CandidateFilter:   /input/Godzilla (USA).zip
[21:24:46.020] WARN:  CandidateFilter:   /input/Faceball (Japan) (Demo).zip
[21:24:46.020] WARN:  CandidateFilter:   /input/Faceball (Japan).zip
[21:24:46.020] WARN:  CandidateFilter:   /input/Buster Bros. (USA).zip
[21:24:46.186] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip:
[21:24:46.186] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip
[21:24:46.186] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.186] WARN:  CandidateFilter:   /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:50.495] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
A).zip:
[21:24:50.495] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:50.495] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN:  CandidateFilter:   /input/Neo Nectaris (Japan).zip
[21:24:50.495] WARN:  CandidateFilter:   /input/Blood Gear (Japan).zip
[21:24:54.269] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
B).zip:
[21:24:54.269] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc B).zip
[21:24:54.269] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:54.269] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.399] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
A).zip:
[21:26:02.399] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:26:02.399] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.399] WARN:  CandidateFilter:   /input/Neo Nectaris (Japan).zip
[21:26:02.399] WARN:  CandidateFilter:   /input/Blood Gear (Japan).zip
[21:26:02.533] WARN:  CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
B).zip:
[21:26:02.533] WARN:  CandidateFilter:   /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.533] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.533] WARN:  CandidateFilter:   /input/Dungeon Master - Theron's Quest (USA).zip
[21:26:22.963] WARN:  CandidateFilter: Cannot move different files to: /output/Godzilla (USA).zip:
[21:26:22.963] WARN:  CandidateFilter:   /input/Godzilla (USA).zip
[21:26:22.963] WARN:  CandidateFilter:   /input/Buster Bros. (USA).zip
[21:26:54.580] WARN:  CandidateFilter: Cannot move different files to: /output/TV Show, The (Japan).zip:
[21:26:54.580] WARN:  CandidateFilter:   /input/TV Show, The (Japan).zip
[21:26:54.580] WARN:  CandidateFilter:   /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:27:24.564] WARN:  CandidateFilter: Cannot move different files to: /output/Gain Ground SX (Japan).zip:
[21:27:24.564] WARN:  CandidateFilter:   /input/Gain Ground SX (Japan).zip
[21:27:24.564] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:27:44.807] WARN:  CandidateFilter: Cannot move different files to: /output/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo)
(SABS).zip:
[21:27:44.807] WARN:  CandidateFilter:   /input/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo) (SABS).zip
[21:27:44.807] WARN:  CandidateFilter:   /input/Atlas, The - Renaissance Voyager (Japan).zip
emmercm commented 1 year ago

@chainsawsalad let's dig in on one of the examples and I can make a unit test around it.

For this one:

[21:20:12.938] WARN:  CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN:  CandidateFilter:   /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN:  CandidateFilter:   /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip

I see a shared track 1 again (SHA1 3b472f5e80bbf8259e09f8e4f547567bddb02490).

Could you list the contents of those input zips? Something that includes name, size, and CRC32 such as:

unzip -vl <file.zip>
chainsawsalad commented 1 year ago

@emmercm does this get you what you need?

Archive:  Zero 4 Champ II (Japan).zip
TORRENTZIPPED-0C146E99
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
 7916832  Defl:X  4641426  41% 12-24-1996 23:32 22144d0f  Zero 4 Champ II (Japan) (Track 01).bin
25542720  Defl:X 13182349  48% 12-24-1996 23:32 a3ecc4eb  Zero 4 Champ II (Japan) (Track 02).bin
13041840  Defl:X 11439731  12% 12-24-1996 23:32 5f4717be  Zero 4 Champ II (Japan) (Track 03).bin
11741184  Defl:X 10545042  10% 12-24-1996 23:32 a929f0e1  Zero 4 Champ II (Japan) (Track 04).bin
11475408  Defl:X 10249938  11% 12-24-1996 23:32 8ce00863  Zero 4 Champ II (Japan) (Track 05).bin
10151232  Defl:X  9223726   9% 12-24-1996 23:32 1b052c36  Zero 4 Champ II (Japan) (Track 06).bin
13135920  Defl:X 11843290  10% 12-24-1996 23:32 54d7853c  Zero 4 Champ II (Japan) (Track 07).bin
24213840  Defl:X 21816717  10% 12-24-1996 23:32 aad2c61a  Zero 4 Champ II (Japan) (Track 08).bin
11799984  Defl:X 10449265  11% 12-24-1996 23:32 10f0d49b  Zero 4 Champ II (Japan) (Track 09).bin
16334640  Defl:X 14788157  10% 12-24-1996 23:32 9eb8f90f  Zero 4 Champ II (Japan) (Track 10).bin
11338992  Defl:X 10188609  10% 12-24-1996 23:32 fa3d048a  Zero 4 Champ II (Japan) (Track 11).bin
12084576  Defl:X 10836032  10% 12-24-1996 23:32 33167322  Zero 4 Champ II (Japan) (Track 12).bin
10513440  Defl:X  9508049  10% 12-24-1996 23:32 09fb1bd2  Zero 4 Champ II (Japan) (Track 13).bin
16325232  Defl:X 14875132   9% 12-24-1996 23:32 7404385c  Zero 4 Champ II (Japan) (Track 14).bin
15266832  Defl:X 13763972  10% 12-24-1996 23:32 7f2b9388  Zero 4 Champ II (Japan) (Track 15).bin
11153184  Defl:X  9604136  14% 12-24-1996 23:32 0c360c9b  Zero 4 Champ II (Japan) (Track 16).bin
11148480  Defl:X 10174000   9% 12-24-1996 23:32 2f712235  Zero 4 Champ II (Japan) (Track 17).bin
11357808  Defl:X 10320677   9% 12-24-1996 23:32 8a7b39b2  Zero 4 Champ II (Japan) (Track 18).bin
 7067760  Defl:X  6152489  13% 12-24-1996 23:32 616c4702  Zero 4 Champ II (Japan) (Track 19).bin
13279392  Defl:X 11852326  11% 12-24-1996 23:32 27706096  Zero 4 Champ II (Japan) (Track 20).bin
 7923888  Defl:X  6785898  14% 12-24-1996 23:32 fecb096f  Zero 4 Champ II (Japan) (Track 21).bin
14888160  Defl:X 13536732   9% 12-24-1996 23:32 9528018f  Zero 4 Champ II (Japan) (Track 22).bin
 6385680  Defl:X  5609726  12% 12-24-1996 23:32 26c2432a  Zero 4 Champ II (Japan) (Track 23).bin
16367568  Defl:X 14766983  10% 12-24-1996 23:32 ce330910  Zero 4 Champ II (Japan) (Track 24).bin
  846720  Defl:X   717964  15% 12-24-1996 23:32 11bf5dbd  Zero 4 Champ II (Japan) (Track 25).bin
 3281040  Defl:X  2933325  11% 12-24-1996 23:32 f9188f3a  Zero 4 Champ II (Japan) (Track 26).bin
  846720  Defl:X   717964  15% 12-24-1996 23:32 11bf5dbd  Zero 4 Champ II (Japan) (Track 27).bin
 4887456  Defl:X  4323950  12% 12-24-1996 23:32 8d2dbe56  Zero 4 Champ II (Japan) (Track 28).bin
12945408  Defl:X 11156588  14% 12-24-1996 23:32 8d234a26  Zero 4 Champ II (Japan) (Track 29).bin
17611776  Defl:X 15340413  13% 12-24-1996 23:32 92ee3fb4  Zero 4 Champ II (Japan) (Track 30).bin
21073920  Defl:X 18202928  14% 12-24-1996 23:32 7b03db99  Zero 4 Champ II (Japan) (Track 31).bin
14556528  Defl:X 13229124   9% 12-24-1996 23:32 1f3057c2  Zero 4 Champ II (Japan) (Track 32).bin
10520496  Defl:X  9323887  11% 12-24-1996 23:32 887d8c2a  Zero 4 Champ II (Japan) (Track 33).bin
11219040  Defl:X  9843611  12% 12-24-1996 23:32 05d3e3a4  Zero 4 Champ II (Japan) (Track 34).bin
12411504  Defl:X 10909543  12% 12-24-1996 23:32 b4cabb3b  Zero 4 Champ II (Japan) (Track 35).bin
15285648  Defl:X 13532102  12% 12-24-1996 23:32 8f997117  Zero 4 Champ II (Japan) (Track 36).bin
11284896  Defl:X  9755615  14% 12-24-1996 23:32 798a830f  Zero 4 Champ II (Japan) (Track 37).bin
24498432  Defl:X 21884726  11% 12-24-1996 23:32 ac1cbe07  Zero 4 Champ II (Japan) (Track 38).bin
19533360  Defl:X 17334422  11% 12-24-1996 23:32 36df83e6  Zero 4 Champ II (Japan) (Track 39).bin
21897120  Defl:X 19981346   9% 12-24-1996 23:32 137e2970  Zero 4 Champ II (Japan) (Track 40).bin
 2629536  Defl:X  1992860  24% 12-24-1996 23:32 d02a90b7  Zero 4 Champ II (Japan) (Track 41).bin
14224896  Defl:X 12706215  11% 12-24-1996 23:32 aa345590  Zero 4 Champ II (Japan) (Track 42).bin
25192272  Defl:X 13171863  48% 12-24-1996 23:32 661f8f8e  Zero 4 Champ II (Japan) (Track 43).bin
    4187  Defl:X      359  91% 12-24-1996 23:32 a8c5c66e  Zero 4 Champ II (Japan).cue
--------          -------  ---                            -------
555205547         473213207  15%                            44 files
Archive:  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
TORRENTZIPPED-00CC60E5
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
 7916832  Defl:X  4641426  41% 12-24-1996 23:32 22144d0f  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 1).bin
10536960  Defl:X  2477771  77% 12-24-1996 23:32 59cc5f3f  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 2).bin
 7785120  Defl:X  6782271  13% 12-24-1996 23:32 877dde76  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 3).bin
18992400  Defl:X 17376419   9% 12-24-1996 23:32 3236adb9  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 4).bin
 9252768  Defl:X  8439144   9% 12-24-1996 23:32 341cc45f  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 5).bin
27854736  Defl:X 25710412   8% 12-24-1996 23:32 e3646dc1  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 6).bin
 5216736  Defl:X  4804265   8% 12-24-1996 23:32 e3792471  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 7).bin
     984  Defl:X      202  80% 12-24-1996 23:32 54a01e7d  Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).cue
--------          -------  ---                            -------
87556536         70231910  20%                            8 files
emmercm commented 1 year ago

@chainsawsalad that's perfect, thank you! I was hoping there was something easy to spot in the output, but everything checks out.

I've been spending all evening staring at this, and the "how" is nitty gritty, but the "why" is because some of these games have duplicate tracks within them:

<rom name="Zero 4 Champ II (Japan) (Track 25).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>
<!-- ... -->
<rom name="Zero 4 Champ II (Japan) (Track 27).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>

I'm still hacking away at it.

emmercm commented 1 year ago

The unit tests aren't passing, I need to clean them up, but feel free to try again for a sneak peek @chainsawsalad 👍

chainsawsalad commented 1 year ago

@emmercm just re-ran my on my whole PC Engine CD collection--no errors!

github-actions[bot] commented 1 year ago

:lock: Inactive issue lock

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.