Closed chainsawsalad closed 1 year ago
I'm sorry @chainsawsalad, I had read this and had been thinking about it, and then never got back to you.
This is because a number of PC Engine CD discs share the same track 1 it looks like, with the SHA1 e1baaf01eee215def8bd6f6aa080e3a41ffc74f4
. Internally, igir
de-duplicates input files, because the thought was it doesn't particularly matter where a file of a certain hash comes from. But in the case of moving games with multiple files (e.g. discs), we do kind of care, and would prefer the files come from the same archive.
This is going to take quite a bit of internal refactoring to tackle, to break that input file assumption. I'll see what I can do!
On the logging, I agree with you, that's really aggressive. I'll drop the level down to DEBUG.
Hi @chainsawsalad, I've been hacking at this slowly over the past few weeks, and I think I might have the solution.
Could you help me by running the testing instructions at https://github.com/emmercm/igir/pull/467#issuecomment-1629761257 against your local setup? I spent some time being able to run igir
from a GitHub branch/PR.
@emmercm just tried it as i ran in the same issue for some ps games. Works like a charm for those games.
@emmercm I re-ran my test example with the original two ROMs, and things look good, however I ran it against the rest of my library and I do still see examples of this happening with other ROMs.
Working example:
npm exec --yes -- "github:emmercm/igir#emmercm/same-input-file-for-everything" -vvv move test report --dat ../../DATs/Redump/NEC\ -\ PC\ Engine\ CD\ \&\ TurboGrafx\ CD\ -\ Datfile\ \(541\)\ \(2023-04-09\ 22-11-42\).dat --input ./input --output ./output
> ls -l ./output/
total 514436
-rwxrwxrwx 1 jeff users 368156053 May 20 13:28 'Fighting Street (USA).zip'
-rwxrwxrwx+ 1 jeff users 158625630 May 20 13:23 'Jack Nicklaus Turbo Golf (USA).zip'
Not working examples, extracted out of command output:
[21:20:12.938] WARN: CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN: CandidateFilter: /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:27.716] WARN: CandidateFilter: Cannot move different files to: /output/Bonanza Bros. (Japan).zip:
[21:21:27.716] WARN: CandidateFilter: /input/Bonanza Bros. (Japan).zip
[21:21:27.716] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:21:30.518] WARN: CandidateFilter: Cannot move different files to: /output/Record of Lodoss War II (Japan) (Demo).zip:
[21:21:30.518] WARN: CandidateFilter: /input/Record of Lodoss War II (Japan) (Demo).zip
[21:21:30.518] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:21:32.957] WARN: CandidateFilter: Cannot move different files to: /output/Faceball (Japan) (Demo).zip:
[21:21:32.957] WARN: CandidateFilter: /input/Faceball (Japan) (Demo).zip
[21:21:32.957] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:22:13.799] WARN: CandidateFilter: Cannot move different files to: /output/Travel Eple (Japan).zip:
[21:22:13.799] WARN: CandidateFilter: /input/Travel Eple (Japan).zip
[21:22:13.799] WARN: CandidateFilter: /input/Gradius II - Gofer no Yabou (Japan).zip
[21:23:54.717] WARN: CandidateFilter: Cannot move different files to: /output/Martial Champion (Japan).zip:
[21:23:54.717] WARN: CandidateFilter: /input/Martial Champion (Japan).zip
[21:23:54.717] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:23:54.717] WARN: CandidateFilter: /input/Fighting Street (Japan) (Rev 5).zip
[21:24:37.618] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip:
[21:24:37.618] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 2 - 1993 Spring (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/CDI Karaoke 10 (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM (Japan).zip
[21:24:37.618] WARN: CandidateFilter: /input/Lords of Thunder (USA).zip
[21:24:37.618] WARN: CandidateFilter: /input/Chou Eiyuu Densetsu - Dynastic Hero (Japan).zip
[21:24:39.419] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip:
[21:24:39.419] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 3 - 1993 Summer (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:39.419] WARN: CandidateFilter: /input/CDI Karaoke 10 (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Might and Magic III - Isles of Terra (Japan).zip
[21:24:39.419] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:24:46.020] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip:
[21:24:46.020] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 4 - 1993 Winter (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Godzilla (USA).zip
[21:24:46.020] WARN: CandidateFilter: /input/Faceball (Japan) (Demo).zip
[21:24:46.020] WARN: CandidateFilter: /input/Faceball (Japan).zip
[21:24:46.020] WARN: CandidateFilter: /input/Buster Bros. (USA).zip
[21:24:46.186] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip:
[21:24:46.186] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 5 - 1994 Spring (Japan).zip
[21:24:46.186] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:46.186] WARN: CandidateFilter: /input/Doukyuusei (Japan) (Rev 3).zip
[21:24:50.495] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
A).zip:
[21:24:50.495] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:50.495] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:24:50.495] WARN: CandidateFilter: /input/Neo Nectaris (Japan).zip
[21:24:50.495] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:24:54.269] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc
B).zip:
[21:24:54.269] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM 6 - 1994 Summer (Japan) (Disc B).zip
[21:24:54.269] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:24:54.269] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.399] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
A).zip:
[21:26:02.399] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc A).zip
[21:26:02.399] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.399] WARN: CandidateFilter: /input/Neo Nectaris (Japan).zip
[21:26:02.399] WARN: CandidateFilter: /input/Blood Gear (Japan).zip
[21:26:02.533] WARN: CandidateFilter: Cannot move different files to: /output/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc
B).zip:
[21:26:02.533] WARN: CandidateFilter: /input/PC Engine Hyper Catalog CD-ROM - 1994 Summer (Japan) (Disc B).zip
[21:26:02.533] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:26:02.533] WARN: CandidateFilter: /input/Dungeon Master - Theron's Quest (USA).zip
[21:26:22.963] WARN: CandidateFilter: Cannot move different files to: /output/Godzilla (USA).zip:
[21:26:22.963] WARN: CandidateFilter: /input/Godzilla (USA).zip
[21:26:22.963] WARN: CandidateFilter: /input/Buster Bros. (USA).zip
[21:26:54.580] WARN: CandidateFilter: Cannot move different files to: /output/TV Show, The (Japan).zip:
[21:26:54.580] WARN: CandidateFilter: /input/TV Show, The (Japan).zip
[21:26:54.580] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
[21:27:24.564] WARN: CandidateFilter: Cannot move different files to: /output/Gain Ground SX (Japan).zip:
[21:27:24.564] WARN: CandidateFilter: /input/Gain Ground SX (Japan).zip
[21:27:24.564] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
[21:27:44.807] WARN: CandidateFilter: Cannot move different files to: /output/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo)
(SABS).zip:
[21:27:44.807] WARN: CandidateFilter: /input/Garou Densetsu 2 - Aratanaru Tatakai (Japan) (Demo) (SABS).zip
[21:27:44.807] WARN: CandidateFilter: /input/Atlas, The - Renaissance Voyager (Japan).zip
@chainsawsalad let's dig in on one of the examples and I can make a unit test around it.
For this one:
[21:20:12.938] WARN: CandidateFilter: Cannot move different files to: /output/Zero 4 Champ II (Japan).zip:
[21:20:12.938] WARN: CandidateFilter: /input/Zero 4 Champ II (Japan).zip
[21:20:12.938] WARN: CandidateFilter: /input/Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
I see a shared track 1 again (SHA1 3b472f5e80bbf8259e09f8e4f547567bddb02490
).
Could you list the contents of those input zips? Something that includes name, size, and CRC32 such as:
unzip -vl <file.zip>
@emmercm does this get you what you need?
Archive: Zero 4 Champ II (Japan).zip
TORRENTZIPPED-0C146E99
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
7916832 Defl:X 4641426 41% 12-24-1996 23:32 22144d0f Zero 4 Champ II (Japan) (Track 01).bin
25542720 Defl:X 13182349 48% 12-24-1996 23:32 a3ecc4eb Zero 4 Champ II (Japan) (Track 02).bin
13041840 Defl:X 11439731 12% 12-24-1996 23:32 5f4717be Zero 4 Champ II (Japan) (Track 03).bin
11741184 Defl:X 10545042 10% 12-24-1996 23:32 a929f0e1 Zero 4 Champ II (Japan) (Track 04).bin
11475408 Defl:X 10249938 11% 12-24-1996 23:32 8ce00863 Zero 4 Champ II (Japan) (Track 05).bin
10151232 Defl:X 9223726 9% 12-24-1996 23:32 1b052c36 Zero 4 Champ II (Japan) (Track 06).bin
13135920 Defl:X 11843290 10% 12-24-1996 23:32 54d7853c Zero 4 Champ II (Japan) (Track 07).bin
24213840 Defl:X 21816717 10% 12-24-1996 23:32 aad2c61a Zero 4 Champ II (Japan) (Track 08).bin
11799984 Defl:X 10449265 11% 12-24-1996 23:32 10f0d49b Zero 4 Champ II (Japan) (Track 09).bin
16334640 Defl:X 14788157 10% 12-24-1996 23:32 9eb8f90f Zero 4 Champ II (Japan) (Track 10).bin
11338992 Defl:X 10188609 10% 12-24-1996 23:32 fa3d048a Zero 4 Champ II (Japan) (Track 11).bin
12084576 Defl:X 10836032 10% 12-24-1996 23:32 33167322 Zero 4 Champ II (Japan) (Track 12).bin
10513440 Defl:X 9508049 10% 12-24-1996 23:32 09fb1bd2 Zero 4 Champ II (Japan) (Track 13).bin
16325232 Defl:X 14875132 9% 12-24-1996 23:32 7404385c Zero 4 Champ II (Japan) (Track 14).bin
15266832 Defl:X 13763972 10% 12-24-1996 23:32 7f2b9388 Zero 4 Champ II (Japan) (Track 15).bin
11153184 Defl:X 9604136 14% 12-24-1996 23:32 0c360c9b Zero 4 Champ II (Japan) (Track 16).bin
11148480 Defl:X 10174000 9% 12-24-1996 23:32 2f712235 Zero 4 Champ II (Japan) (Track 17).bin
11357808 Defl:X 10320677 9% 12-24-1996 23:32 8a7b39b2 Zero 4 Champ II (Japan) (Track 18).bin
7067760 Defl:X 6152489 13% 12-24-1996 23:32 616c4702 Zero 4 Champ II (Japan) (Track 19).bin
13279392 Defl:X 11852326 11% 12-24-1996 23:32 27706096 Zero 4 Champ II (Japan) (Track 20).bin
7923888 Defl:X 6785898 14% 12-24-1996 23:32 fecb096f Zero 4 Champ II (Japan) (Track 21).bin
14888160 Defl:X 13536732 9% 12-24-1996 23:32 9528018f Zero 4 Champ II (Japan) (Track 22).bin
6385680 Defl:X 5609726 12% 12-24-1996 23:32 26c2432a Zero 4 Champ II (Japan) (Track 23).bin
16367568 Defl:X 14766983 10% 12-24-1996 23:32 ce330910 Zero 4 Champ II (Japan) (Track 24).bin
846720 Defl:X 717964 15% 12-24-1996 23:32 11bf5dbd Zero 4 Champ II (Japan) (Track 25).bin
3281040 Defl:X 2933325 11% 12-24-1996 23:32 f9188f3a Zero 4 Champ II (Japan) (Track 26).bin
846720 Defl:X 717964 15% 12-24-1996 23:32 11bf5dbd Zero 4 Champ II (Japan) (Track 27).bin
4887456 Defl:X 4323950 12% 12-24-1996 23:32 8d2dbe56 Zero 4 Champ II (Japan) (Track 28).bin
12945408 Defl:X 11156588 14% 12-24-1996 23:32 8d234a26 Zero 4 Champ II (Japan) (Track 29).bin
17611776 Defl:X 15340413 13% 12-24-1996 23:32 92ee3fb4 Zero 4 Champ II (Japan) (Track 30).bin
21073920 Defl:X 18202928 14% 12-24-1996 23:32 7b03db99 Zero 4 Champ II (Japan) (Track 31).bin
14556528 Defl:X 13229124 9% 12-24-1996 23:32 1f3057c2 Zero 4 Champ II (Japan) (Track 32).bin
10520496 Defl:X 9323887 11% 12-24-1996 23:32 887d8c2a Zero 4 Champ II (Japan) (Track 33).bin
11219040 Defl:X 9843611 12% 12-24-1996 23:32 05d3e3a4 Zero 4 Champ II (Japan) (Track 34).bin
12411504 Defl:X 10909543 12% 12-24-1996 23:32 b4cabb3b Zero 4 Champ II (Japan) (Track 35).bin
15285648 Defl:X 13532102 12% 12-24-1996 23:32 8f997117 Zero 4 Champ II (Japan) (Track 36).bin
11284896 Defl:X 9755615 14% 12-24-1996 23:32 798a830f Zero 4 Champ II (Japan) (Track 37).bin
24498432 Defl:X 21884726 11% 12-24-1996 23:32 ac1cbe07 Zero 4 Champ II (Japan) (Track 38).bin
19533360 Defl:X 17334422 11% 12-24-1996 23:32 36df83e6 Zero 4 Champ II (Japan) (Track 39).bin
21897120 Defl:X 19981346 9% 12-24-1996 23:32 137e2970 Zero 4 Champ II (Japan) (Track 40).bin
2629536 Defl:X 1992860 24% 12-24-1996 23:32 d02a90b7 Zero 4 Champ II (Japan) (Track 41).bin
14224896 Defl:X 12706215 11% 12-24-1996 23:32 aa345590 Zero 4 Champ II (Japan) (Track 42).bin
25192272 Defl:X 13171863 48% 12-24-1996 23:32 661f8f8e Zero 4 Champ II (Japan) (Track 43).bin
4187 Defl:X 359 91% 12-24-1996 23:32 a8c5c66e Zero 4 Champ II (Japan).cue
-------- ------- --- -------
555205547 473213207 15% 44 files
Archive: Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).zip
TORRENTZIPPED-00CC60E5
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
7916832 Defl:X 4641426 41% 12-24-1996 23:32 22144d0f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 1).bin
10536960 Defl:X 2477771 77% 12-24-1996 23:32 59cc5f3f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 2).bin
7785120 Defl:X 6782271 13% 12-24-1996 23:32 877dde76 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 3).bin
18992400 Defl:X 17376419 9% 12-24-1996 23:32 3236adb9 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 4).bin
9252768 Defl:X 8439144 9% 12-24-1996 23:32 341cc45f Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 5).bin
27854736 Defl:X 25710412 8% 12-24-1996 23:32 e3646dc1 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 6).bin
5216736 Defl:X 4804265 8% 12-24-1996 23:32 e3792471 Adventure Quiz - Capcom World + Hatena no Daibouken (Japan) (Track 7).bin
984 Defl:X 202 80% 12-24-1996 23:32 54a01e7d Adventure Quiz - Capcom World + Hatena no Daibouken (Japan).cue
-------- ------- --- -------
87556536 70231910 20% 8 files
@chainsawsalad that's perfect, thank you! I was hoping there was something easy to spot in the output, but everything checks out.
I've been spending all evening staring at this, and the "how" is nitty gritty, but the "why" is because some of these games have duplicate tracks within them:
<rom name="Zero 4 Champ II (Japan) (Track 25).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>
<!-- ... -->
<rom name="Zero 4 Champ II (Japan) (Track 27).bin" size="846720" crc="11bf5dbd" md5="b0845f9d312f9686079470c88cb62596" sha1="9f5fd5f2788d144e0eb61e2edfbbad0deb090350"/>
I'm still hacking away at it.
The unit tests aren't passing, I need to clean them up, but feel free to try again for a sneak peek @chainsawsalad 👍
@emmercm just re-ran my on my whole PC Engine CD collection--no errors!
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Paste the command
npx -yes igir@latest -vvv move test report --dat ../DATs/Redump/NEC\ -\ PC\ Engine\ CD\ \&\ TurboGrafx\ CD\ -\ Datfile\ \(541\)\ \(2023-04-09\ 22-11-42\).dat --input ./input --output ./output
Describe the bug
Certain optical media files seem to behave unexpectedly when processed against certain other optical media files. I haven't been able to figure out why this happens, but I've isolated many examples of it. It results in a failure to
copy
ormove
one or both files.In the below example
igir
fails to identifyJack Nicklaus Turbo Golf (USA).zip
as a candidate when it should, and it spits out the following warning:Note the conflict with
Fighting Street (USA).zip
. This file is later successfully and properly identified as a candidate and is moved or copied as such. If I removeFighting Street (USA).zip
from./input/
,Jack Nicklaus Turbo Golf (USA).zip
is properly identified as a candidate and the tool behaves as expected. I cannot reproduce this behavior with any other combination of files againstJack Nicklaus Turbo Golf (USA).zip
.Here are is a very short, incomplete list of other conflicts I've encountered (just TurboGrafx, but I see this for many other optical-based system DATs as well):
More details:
Excerpt from
NEC - PC Engine CD & TurboGrafx CD - Datfile (541) (2023-04-09 22-11-42).dat
:Hashes on disk (they match the dat file):
Expected behavior
I expect otherwise positive matching candidates to be treated as such, and not conflict with anything else going on.
The log message
let message = `Cannot ${this.options.writeString()} different files to: ${duplicateOutput}:`;
is unintuitive and I honestly don't understand what it means when this issue is encountered. Maybe clearer messaging would help?Unrelated to this behavior specifically, but the console is flooded with
missing XX files for: <unrelated ROM name>
. In my example of using only two ROMs in./input/
, I see dozens of warnings of what look like attempts to match one of the ROMs to the wrong candidate, and the tool being overly helpful in explaining why. For example:This is true, but
Ys Book I & II (USA)
is not among the ROMs in./input/
. This doesn't feel like it rises to the occasion of needing aWARN
, especially because it's not doing this for every ROM in the DAT, just a subset. See the attached output for a better overview.Debug logs
log.txt
igir version
1.6.4
Node.js version
18.12.0
Operating system
Linux
Additional context
No response