mtgjson / mtgsqlive

MTGJSON build scripts to generate alternative data formats
https://mtgjson.com
MIT License
54 stars 32 forks source link

Duplicate Entries in setBoosterSheetCards and setBoosterSheets #96

Closed dan-kez closed 1 year ago

dan-kez commented 1 year ago

Hi all! I've been using the sqlite and noticed that there seem to be duplicates in setBoosterSheetCards. For example

SELECT
    *
FROM
    setBoosterSheetCards
WHERE
    cardUuid = "04ef11e8-3918-5332-9f7c-3dc29d972c5e"
    and setCode = "CMM"
    and sheetName = "rareMythicShowcase"

Returns

04ef11e8-3918-5332-9f7c-3dc29d972c5e    1    CMM    rareMythicShowcase
04ef11e8-3918-5332-9f7c-3dc29d972c5e    1    CMM    rareMythicShowcase

I'm curious if this is intentional in that the same card can show multiple times on a sheet or if this is an error where this should have been aggregated to a cardweight of 2.

Comparably, the following query returns 18376 rows where all values are duplicated in the table. These seem to be across multiple sets as well.

SELECT
    count(*)
FROM
    setBoosterSheetCards
GROUP BY
    cardUuid,
    cardWeight,
    setCode,
    sheetName
HAVING
    count(*) > 1

Using a similar query it seems there may be a similar issue in setBoosterSheets with 272 duplicate entries.

I posted this in the discord and was advised to forward the details to a github issue.


Edit:

I think I may have partially figured out the challenge here. sheetName + setCode is not sufficiently unique to differentiate sheets between packs.

For example, in CMM f2d7ee8f-f031-5fda-ae70-2ca615da2989 is present in both default and collector under the same sheetName nonlegendaryRareMythic

The current schema (shown below) is not sufficient to disambiguate sheets between boosters.

                            "sheetName": sheet_name,
                            "cardUuid": card_uuid,
                            "cardWeight": card_weight,

It is likely required that boosterName also be added to this schema OR a foreign key relationship with a primary key is created with setBoosterContents.