Incorrect Region Priority - Asia > Japan?

retroNUC commented 6 months ago

Describe the bug Seeing some instances on Sony PlayStation Vita where "Asia" region is taking priority over "Japan" due to language preference, despite no language data existing in dats/metadata.

Expected behavior Region order should be respected (rather than assuming (En) language on an "Asia" region release?)

Operating system - Windows 11 Retool edition - retool-gui.exe Retool version - 2.01.8

DAT data (no language in name):

<game name="Gundam Breaker 2 (Japan)" id="09291">
    <description>Gundam Breaker 2 (Japan)</description>
    <rom name="zInmXKZWTIZNNmGwCvCaJHpIbcIwqJotFneYeuvSgbnjLjYZKasxzZDdxLgaigXioURrDQixdXGeEDLVDuRQTSDmJTSPGTOxXCeBv.pkg" size="3536207312" crc="b33d8ba2" md5="ff842843b1a4c974cd3e7a44eb5b24f5" sha1="1cd34c2564947edb171306c35b9ca418db41c28f"/>
</game>
<game name="Gundam Breaker 2 (Asia)" id="07907">
    <description>Gundam Breaker 2 (Asia)</description>
    <rom name="HP0700-PCSH00132_00-GUNDAMBREAKER2VM_bg_1_bd02636731d7b08a5e1d32c5bd13cf01dd304fc2.pkg" size="3539095216" crc="cee2017c" md5="00f3bc4da12631fd849f8fecfecf28c9" sha1="597dc230b739ecbe43f8e227d9651faa220df4e4"/>
</game>

Region Settings ("Japan" priority over "Asia"):

Trace Results:

Retool 2.01.8
-------------
* Reading DAT file: "C:\Emulation\tools\retool_v2\dats\Sony - PlayStation Vita (PSN)
  (Content) (20231009-113000) (Games).dat"
* Validating DAT file... done.

|  DAT DETAILS
|  Description: Sony - PlayStation Vita (PSN) (Content)
|  Author/s: [mRg], bsbt, buckwheat, FakeShemp, Gefflon, Hiccup, Landcross, norkmetnoil577,
   SonGoku
|  URL: https://www.no-intro.org
|  Version: 20231009-113000
|  Numbered dat: No

* Processing DAT file... done.
* Applying clone list categories... done.
* Analyzing clone list variants... done.
* Applying category and user excludes...

Stage: Exclusions

REF0042: ACTION: Excluded due to categories: Demos

* Gundam Breaker 2 (Asia) (Demo)

Press enter to continue

REF0042: ACTION: Excluded due to categories: Demos

* Gundam Breaker 2 - United Front (Japan) (Demo)

Press enter to continue

done.
* Removing titles without specified regions... done.
* Selecting 1G1R titles...

Stage: Parent selection
Group: gundam breaker 2

Region: Japan

REF0074: [gundam breaker 2] Only one title in this region, skipping filtering:

* Gundam Breaker 2 (Japan)

Press enter to continue

Region: Asia

REF0074: [gundam breaker 2] Only one title in this region, skipping filtering:

* Gundam Breaker 2 (Asia)

Press enter to continue

Region: All

REF0013: [gundam breaker 2] Cross region parents:

* Gundam Breaker 2 (Asia)
* Gundam Breaker 2 (Japan)

Press enter to continue

REF0015: [gundam breaker 2] Highest language priority titles:

Highest language priority per short name:
* gundam breaker 2 | (0) En(?:-[A-Z][A-Z])?

Titles that match the highest language priority:
+ Keeping: (En)                            [gundam breaker 2] Gundam Breaker 2 (Asia)
- Removing: (Ja)                           [gundam breaker 2] Gundam Breaker 2 (Japan)

Press enter to continue

REF0014: [gundam breaker 2] Group after lower priority region, non-superset clones removed:

* Gundam Breaker 2 (Asia)

Press enter to continue

REF0018: [gundam breaker 2] Group after production title check:

* Gundam Breaker 2 (Asia)

Press enter to continue

REF0019: [gundam breaker 2] ACTION: Clone assignments (might be reassigned later):

+ Gundam Breaker 2 (Asia) is the 1G1R title
- Gundam Breaker 2 (Japan) is a clone of Gundam Breaker 2 (Asia)

Press enter to continue
* Selecting 1G1R titles... done.
* Analyzing clone list variants... done.
* Applying user includes...

Stage: Inclusions
done.

Stage: Post filters
* Trace complete.

unexpectedpanda commented 6 months ago

Languages are prioritized over regions. That's working as intended and is largely responsible for a lot of the better choices v2 makes over v1. You can reverse that with the Prefer regions over languages option.

There's nothing in the DAT or metadata for the title, but Retool usually sets a single implied language for a region as a backup -- essentially an educated guess if a title doesn't explicitly list its languages anywhere. There are a few regions where there are no implied languages -- Scandinavia for example -- but the more of those there are, the more problems occur during comparisons. I toyed with the idea of including multiple implied languages, but the Asia region was the one that scuttled that idea, given the number of languages and the unlikeliness that a title supports, say, Tamil.

Asia was set to an implied language of English (you can see this in config/internal-config.json) because the majority of the Asia titles at the time supported English. I can reassess whether that's still a good choice for an upcoming release.

In the meantime, Asia is a terrible thing to use as a region name without listing language support. Fixing the data at the source would be the best way forward.

unexpectedpanda commented 6 months ago

Okay, I've run the assessment and I'll be removing English as the implied language for Asia in a future release. A few things have happened since the decision was originally made, but the major one is the flood of Chinese language titles that have entered the No-Intro set and been assigned as Asia.

retroNUC commented 6 months ago

I certainly agree with you about Asia region being a bit useless in No-Intro without any corresponding language tags. I'm still waiting for them to process the tickets/corrections I've submitted.

To be honest, I would have also agreed with you that implying English made sense, but it's certainly a tricky one. If I think about it from a language perspective, these are the cases:

If an Asia release has English, 99% of the time it's going to have a western region release with English that's going to supersede it
If a game came out in Asia only, it's always going to be the one chosen, so language doesn't come into it
If a game is Asia/Japan only and we have no language info... the majority of the time, I would guess that it's unlikely that the Asia release has English and the Japan release does not. I can think of a few cases where it has happened, but it's exceedingly rare that kind of localization effort happens while never making it to a western region release.

Most of this is just EastAsiaSoft messing things up for everyone :)

unexpectedpanda commented 6 months ago

I'm still waiting for them to process the tickets/corrections I've submitted.

It's not a given thing they'll get to it. Fingers crossed. Each system tends to stay in the hands of a few maintainers, and if they're gone or not interested, things can stall. I'm honestly not sure how effective their ticket system is either, I don't think there's a queue or notification system.

If I think about it from a language perspective, these are the cases...

This tracks for me. Removing the implied English language should generally give the needed effect... except for those who rank Asia above all the constituent regions like Japan, China, Taiwan, Hong Kong, Singapore, India and so on. I suspect that's very few people, but we'll find out.

unexpectedpanda commented 5 months ago

Finally had time to get v2.02 out and also fix this along the way. Thanks for your patience!

unexpectedpanda / retool

Incorrect Region Priority - Asia > Japan? #297