utterances-bot commented 1 year ago

Getting Halo Infinite Match Stats With Official Halo API · Den Delimarsky

Getting visibility into your match performance outside the game.

https://den.dev/blog/halo-api-match-stats/

acurtis166 commented 1 year ago

Hey, Den. First of all, thanks for documenting all your work on the Halo Infinite API. I leaned on it quite a bit as I was developing this project. I was curious if you ever had any luck dissecting the film data referenced in the /spectate response? I've downloaded and decompressed chunks for a sample match, but haven't been able to extract any useful information.

dend commented 1 year ago

@acurtis166 I am working through that in my free time. It's for sure not anything that is related to video content - it's very likely those are data batches fed to the game engine. I wrote a basic parser that goes through the content, but nothing to share yet.

aapokaapo commented 1 year ago

Huge shoutout to both of you guys! I created a discord bot that generates graphs out of the stats pulled from halo using @acurtis166 's SPNKr.py. There's no public repo yet but the API is available at https://halofin.land/docs.

Now I started wondering if the theater recordings could be used to create heatmaps of the player positions during the match and I was so happy to find both of you my heroes talking here about the theater recordings. Are they encrypted or in human readable form?

dend commented 1 year ago

Are they encrypted or in human readable form?

It doesn't look like they are encrypted (if they would be - I wouldn't even bother touching it 😄). The challenge there though is that from my cursory analysis, the data captured there is effectively game engine metadata, which means that reading it without understanding the engine might be more complicated than just reading "Position [X, Y, Z] at time [HH:MM:SS]". That being said, I am actively investigating this!

acurtis166 commented 1 year ago

@aapokaapo Exactly what I was hoping for! Halo 5 public API has an endpoint that returns detailed kill events for a given match which was fun to play with.

aapokaapo commented 12 months ago

I was poking around decompressed recording file from the endpoint. Didn't make much procress since I am highly inexperienced in this, but all I got out is that the file consists of 1033 ComponentObjects such as PlayerWaypointComponent, ManagedObjectNavpointComponent and WeaponStateAmmoComponent to name a few (these components are from the Game Engine I presume). the Component names in the header are 256 bytes long followed by a value or number of somesorts that is 4 bytes long.

I wonder if Halo Forge could help in guessing what a ComponentObject consists of. Looking at Unity or Unreal gamefile structures could probably help too, since I doubt that the Slipspace engine differs that much from them.

dend commented 12 months ago

I was poking around decompressed recording file from the endpoint. Didn't make much procress since I am highly inexperienced in this, but all I got out is that the file consists of 1033 ComponentObjects such as PlayerWaypointComponent, ManagedObjectNavpointComponent and WeaponStateAmmoComponent to name a few (these components are from the Game Engine I presume). the Component names in the header are 256 bytes long followed by a value or number of somesorts that is 4 bytes long.

I wonder if Halo Forge could help in guessing what a ComponentObject consists of. Looking at Unity or Unreal gamefile structures could probably help too, since I doubt that the Slipspace engine differs that much from them.

What's your approach to getting component names, @aapokaapo? For example, ComponentObject or PlayetWaypointComponent.

aapokaapo commented 12 months ago

@dend I opened the file in hex editor and there are 1033 string names readable. I am by no means experienced in this and could be completely in the wrong direction tho...

jackphillips31 commented 8 months ago

Hey Den!

Thank you so much for documenting your experience exploring the Halo Infinite API. I would like to ask if you can provide any insight regarding the paths that are provided within some of the API responses.

Example

In the Match Progression API call you have shown in this blog post part of the response includes strings labeled “Path.” I was wondering if you could share any insight whatsoever regarding how to use these paths.

Endpoint

https://halostats.svc.halowaypoint.com/hi/players/{player}/matches/{matchId}/progression

JSON Response

{
   ”ChallengeProgressState”: [
      {
         "Path": "ChallengeContent/ClientChallengeDefinitions/S1RotationalSet1Challenges/Normal/NTeamSlayerPlay.json"
      }
   ]
}

Questions Regarding Paths

ChallengeContent/ClientChallengeDefinitions/S1RotationalSet1Challenges/Normal/NTeamSlayerPlay.json

What host do I use this path with?
Do different endpoints give paths that have different endpoints or do all endpoints provide paths to the same CDN sort of endpoint?
If they include paths to resources like images, what host do I use with those?
Is it a local path?
Conclusion

I get the feeling like paths like that include paths to resources like Rank Images or the Player Banners. Resources like that are ultimately what I’m after.

Any advice or response is greatly appreciated! Thank you.

aapokaapo commented 8 months ago

@jackphillips31 you should check out a python lib called SPNKr.py, which is based on this blog

jackphillips31 commented 8 months ago

@aapokaapo Thank you for this! I'm creating a React-Native app so I won't actually be able to use this but it has definitely provided some useful information. As far as retrieving images goes, I feel stupid now. I'm not sure how I overlooked the GameCms_GetImage endpoint, but I did. As far as retrieving retrieving information from paths like ChallengeContent/ClientChallengeDefinitions/S1RotationalSet1Challenges/Normal/NTeamSlayerPlay.json I'm still not sure. There is a GameCms_GetItem endpoint so that might be it but I won't know until I get home tonight. I've only briefly looked at SPNKr.py so I'll definitely spend a little more time going over that and seeing if I can gleam anything else from it. Thank you again!

acurtis166 commented 8 months ago

@jackphillips31 Have a look at the following example.

https://gamecms-hacs.svc.halowaypoint.com/hi/Progression/file/ChallengeContent/ClientChallengeDefinitions/S6CapstoneChallenges/CMedalStick.json

Resulting in the response below.

{
    "Description": "Stick an Enemy Spartan with a Grenade",
    "Difficulty": "mythic",
    "Category": "General",
    "Reward": {
        "InventoryItems": [],
        "OperationExperience": 500,
        "InventoryRewards": [
            {
                "InventoryItemPath": "Inventory/armor/coatings/3761780-ArmorCoating-Mark-VII.json",
                "Amount": 1,
                "Type": "ArmorCoating"
            }
        ]
    },
    "ThresholdForSuccess": 1,
    "Title": "Mythic Stick"
}

I was getting 404 when trying for your referenced challenge, but the above worked for me (spartan token authentication required). You can see it is relative to the "Progression/file" sub-path within Halo Infinite CMS.

Questions

What host do I use this path with? It's not always clear, but the game CMS host is a common one for these types of resources.
Do different endpoints give paths that have different endpoints or do all endpoints provide paths to the same CDN sort of endpoint? Again, the CMS host provides many of these, but there are others.
If they include paths to resources like images, what host do I use with those? I'll provide resources for your specific requests.
Is it a local path? Not sure what you mean here.

Rank Images

You'll find the relative paths for all career ranks here:

https://gamecms-hacs.svc.halowaypoint.com/hi/Progression/file/RewardTracks/CareerRanks/careerRank1.json

From which you can construct the url to the actual images (example for "Hero" rank below).

https://gamecms-hacs.svc.halowaypoint.com/hi/Images/file/career_rank/NameplateAdornment/272_Hero.png

If instead you are looking for CSR images, try using the following from waypoint (no auth required).

https://www.halowaypoint.com/images/halo-infinite/csr/platinum_5.png

Player Banners

First, get the appearance info for a player:

https://economy.svc.halowaypoint.com/hi/players/{xuid}/customization/appearance

From there you can build the url to retrieve data for a given customization item (backdrop example below).

https://gamecms-hacs.svc.halowaypoint.com/hi/Progression/file/inventory/spartan/backdropimages/103-000-ui-s2-backgro-5c565f83.json

From there you can build the image url:

https://gamecms-hacs.svc.halowaypoint.com/hi/images/file/progression/backgrounds/ui_s2_background_entrenchedfractureinsignia.png

Hope this was helpful

jackphillips31 commented 8 months ago

@acurtis166

Thank you for your reply!

As I have delved more into the API a lot has become more obvious. At this point I am able to retrieve practically any JSON or any image/image-blob I see. At least I can't think of a single one that I can't retrieve. Except for the example I gave previously which is weird, maybe just a typo? It's unimportant because when I make the same request that resulted in that JSON, the JSON paths in that response I can easily retrieve. It is a little frustrating for something as simple as rebuilding the player banner, you have to make approximately 7 requests just to get the backdrop, banner, and emblem images but I digress.

Another Question...

I'm wondering if there is a dictionary of sorts anywhere where I can gleam information for things like "TeamId" in the response from Stats_GetMatchStats endpoint. Like how "TeamId": 0 is Eagle or Red team and how "TeamId": 1 is Cobra or Blue team. Or how "Outcome": 2 is winner and "Outcome": 3 is loser.

I get the idea...

that sort of stuff, I'm going to have to compile manually. But if there is a way to get definitions for those, please let me know. I know the xml format is more verbose and in some situations (I believe?) it provides strings rather than numbers but as far as for the Stats_GetMatchStats endpoint, it seems to provide numbers for the TeamId and Outcome as well.

Thank you again!

acurtis166 commented 8 months ago

@jackphillips31 This file has many of the enumerated data types at least partially documented. Team and bot names are here.

These are from comparing to match results or from inspecting network traffic while browsing Halo Waypoint. Let us know if you find others!

jackphillips31 commented 8 months ago

@acurtis166 I cannot begin to express my appreciation for this, thank you so much! This is an absolutely fantastic resource. I assume you compiled this yourself? Absolutely great work.

jackphillips31 commented 7 months ago

@acurtis166 I've made a ton of progress on my personal lil project. But I'm getting stuck again, I'll just get straight to the point to see if you have any insights.

TLDR: How do I get the player nameplates and the different colored emblems?

My question is regarding the player's banner/nameplate. I know how to get the backdrop, I know how to get the emblem, I even know how to get the "nameplate adornment" rank images. I'd assume it has something to do with the emblem since in Halo Infinite the emblem and nameplate are one thing sort of? But in the response for the emblem, I'm not seeing anything that is apparently the nameplate.

Economy_PlayerCustomizationAppearance

https://economy.svc.halowaypoint.com/hi/players/{{xuid}}/customization/appearance

{
    "Status": "Success",
    "Appearance": {
        "LastModifiedDateUtc": {
            "ISO8601Date": "2024-04-02T00:10:15.85Z"
        },
        "ActionPosePath": "Inventory/Spartan/ActionPoses/101-000-menu-stance-r-e47e42c4.json",
        "BackdropImagePath": "Inventory/Spartan/BackdropImages/103-000-ui-background-e5fd3381.json",
        "Emblem": {
            "EmblemPath": "Inventory/Spartan/Emblems/104-001-career-rank-0-1e8f3859.json",
            "ConfigurationId": -410600460
        },
        "ServiceTag": "MVP",
        "IntroEmotePath": null,
        "PlayerTitlePath": null
    }
}

GameCms_GetItem

https://gamecms-hacs.svc.halowaypoint.com:443/hi/Progression/file/Inventory/Spartan/Emblems/104-001-olympus-skull-2845463e.json

{
    "TagId": 0,
    "ThemeName": {
        "m_identifier": 626314953
    },
    "EmblemShaderName": {
        "m_identifier": -1
    },
    "AvailableConfigurations": [
        {
            "ConfigurationId": -2101285248,
            "ConfigurationPath": "Configuration/Emblems/Coatings/1000-000-2c95b554.json"
        },
        {
            "ConfigurationId": -1300612690,
            "ConfigurationPath": "Configuration/Emblems/Coatings/1000-000-f37482bf.json"
        },
        {
            "ConfigurationId": 2092759023,
            "ConfigurationPath": "Configuration/Emblems/Coatings/1000-000-f9d7584d.json"
        },
        {
            "ConfigurationId": -1756788076,
            "ConfigurationPath": "Configuration/Emblems/Coatings/1000-000-1de32d39.json"
        },
        {
            "ConfigurationId": 2029654151,
            "ConfigurationPath": "Configuration/Emblems/Coatings/1000-000-715a8962.json"
        }
    ],
    "CommonData": {
        "Id": "104-001-olympus-skull-2845463e",
        "HideUntilOwned": false,
        "Title": {
            "status": "Ready",
            "value": "Skull King",
            "translations": {
                "cs-CZ": "Král lebek",
                "da-DK": "Kraniekonge",
                "de-DE": "Schädelkönig",
                "el-GR": "Βασιλιάς των κρανίων",
                "es-ES": "Rey de las calaveras",
                "es-MX": "Rey de los cráneos",
                "fi-FI": "Kallokuningas",
                "fr-FR": "Crâne couronné",
                "hu-HU": "Koponyakirály",
                "it-IT": "Re teschio",
                "ja-JP": "スカル キング",
                "ko-KR": "해골 왕",
                "nb-NO": "Skallekonge",
                "nl-NL": "Schedelkoning",
                "pl-PL": "Król czaszek",
                "pt-BR": "Rei da Caveira",
                "pt-PT": "Rei Caveira",
                "ru-RU": "Король черепов",
                "sv-SE": "Skallkung",
                "tr-TR": "Kafatası Kralı",
                "zh-CN": "骷髅王",
                "zh-TW": "骷髏王",
                "qps-ploc": "Ѕќůłĺ Кïπĝ !!!",
                "qps-ploca": "Šкϋľľ Κіňğ !!!",
                "qps-plocm": "Ŝĸυļℓ Кΐηĝ !!!"
            }
        },
        "Description": {
            "status": "Ready",
            "value": "Lord of hill and dale.",
            "translations": {
                "cs-CZ": "Pán kopců a údolí.",
                "da-DK": "Herre over bakke og dal.",
                "de-DE": "Herr über Berg und Tal.",
                "el-GR": "Άρχοντας του λόφου και της κοιλάδας.",
                "es-ES": "Señor de la colina y el valle.",
                "es-MX": "Señor de la colina y el valle.",
                "fi-FI": "Kukkulan ja laakson herra.",
                "fr-FR": "Seigneur de la colline et du vallon.",
                "hu-HU": "Dombok-völgyek ura.",
                "it-IT": "Signore della collina e della valle.",
                "ja-JP": "丘と谷の王。",
                "ko-KR": "울퉁불퉁한 땅의 주인입니다.",
                "nb-NO": "Hersker over fjell og daler.",
                "nl-NL": "Heer van heuvel en dal.",
                "pl-PL": "Pan wzgórz i dolin.",
                "pt-BR": "Senhor de colinas e vales.",
                "pt-PT": "Senhor da colina e do vale.",
                "ru-RU": "Владыка холмов и долин.",
                "sv-SE": "Härskare över berg och dal.",
                "tr-TR": "Dere tepenin efendisi.",
                "zh-CN": "山谷之主。",
                "zh-TW": "山丘之主。",
                "qps-ploc": "Łōгď öƒ ĥĩļł ãňď ðдŀė. !!! !!! ",
                "qps-ploca": "£σяδ бƒ ĥĭļł âπð đąľе. !!! !!! ",
                "qps-plocm": "£οřð бƒ нĩļł аņδ ďåļё. !!! !!! "
            }
        },
        "FeatureFlag": true,
        "ItemAvailability": {
            "status": "Test",
            "value": "",
            "translations": {}
        },
        "DateReleased": {
            "ISO8601Date": ""
        },
        "AltName": {
            "status": "Test",
            "value": "104-001-olympus-skull-2845463e",
            "translations": {
                "qps-ploc": "104-001-òŀумφúš-śκΰłŀ-2845463é !!! !!! !!!",
                "qps-ploca": "104-001-ŏŀýmφųѕ-ŝкūłℓ-2845463é !!! !!! !!!",
                "qps-plocm": "104-001-őļўmρΰѕ-ŝķũŀł-2845463з !!! !!! !!!"
            }
        },
        "IconStringId": {
            "m_identifier": -1
        },
        "SpriteBitmap": 0,
        "SpriteFrameIndex": 0,
        "AltSpriteBitmap": 0,
        "AltSpriteFrameIndex": 0,
        "DisplayPath": {
            "Width": 585,
            "Height": 802,
            "Media": {
                "MediaUrl": {
                    "AuthorityId": "",
                    "Path": "progression/Inventory/Emblems/olympus_SkullKing_emblem.png",
                    "RetryPolicyId": "",
                    "TopicName": "",
                    "AcknowledgementTypeId": "NoAcknowledgement",
                    "AuthenticationLifetimeExtensionSupported": false,
                    "ClearanceAware": false
                },
                "MimeType": "",
                "Caption": {
                    "status": "Test",
                    "value": "",
                    "translations": {}
                },
                "AlternateText": {
                    "status": "Test",
                    "value": "",
                    "translations": {}
                },
                "FolderPath": "",
                "FileName": ""
            },
            "MimeType": "image/png",
            "FolderPath": "progression/Inventory/Emblems",
            "FileName": "olympus_SkullKing_emblem.png"
        },
        "Quality": "Common",
        "ManufacturerId": 26,
        "Type": "SpartanEmblem",
        "RewardTrack": "",
        "ParentPaths": [],
        "SortingMetadata": {
            "categoryWeight": 0,
            "subCategoryWeight": 0
        },
        "SeasonNumber": 1,
        "OriginalSeasonNumber": 1,
        "IsCrossCompatible": true,
        "Season": {
            "status": "Test",
            "value": "",
            "translations": {}
        }
    }
}

Configurations

I assume the configurations have to do with the color palette but I tried those and they don't have any useful information either. On the note of the configurations, I'm also wondering how to get the different color emblems/nameplates.

{
    "ConfigurationId": {
        "m_identifier": -1300612690
    },
    "CommonData": {
        "Id": "1000-000-f37482bf",
        "HideUntilOwned": false,
        "Title": {
            "status": "Frozen",
            "value": "Color Palette",
            "translations": {}
        },
        "Description": {
            "status": "Frozen",
            "value": "0",
            "translations": {
                "de-DE": "343other_adrenaline",
                "es-ES": "343other_adrenaline",
                "es-MX": "343other_adrenaline",
                "zh-CN": "343other_adrenaline",
                "zh-TW": "343other_adrenaline",
                "qps-ploc": "343őťћęґ_ăďгэñáĺĩπê !!! !!!",
                "qps-ploca": "343бτĥ℮ř_ªđŗëńăĺĭπё !!! !!!",
                "qps-plocm": "343σŧĥèя_äđřęŋаℓīиè !!! !!!"
            }
        },
        "FeatureFlag": true,
        "ItemAvailability": {
            "status": "Test",
            "value": "",
            "translations": {}
        },
        "DateReleased": {
            "ISO8601Date": ""
        },
        "AltName": {
            "status": "Test",
            "value": "",
            "translations": {}
        },
        "IconStringId": {
            "m_identifier": -1
        },
        "SpriteBitmap": 0,
        "SpriteFrameIndex": 0,
        "AltSpriteBitmap": 0,
        "AltSpriteFrameIndex": 0,
        "DisplayPath": {
            "Width": 355,
            "Height": 327,
            "Media": {
                "MediaUrl": {
                    "AuthorityId": "",
                    "Path": "progression/Inventory/Emblems/emblem_palette.PNG",
                    "RetryPolicyId": "",
                    "TopicName": "",
                    "AcknowledgementTypeId": "NoAcknowledgement",
                    "AuthenticationLifetimeExtensionSupported": false,
                    "ClearanceAware": false
                },
                "MimeType": "",
                "Caption": {
                    "status": "Test",
                    "value": "",
                    "translations": {}
                },
                "AlternateText": {
                    "status": "Test",
                    "value": "",
                    "translations": {}
                },
                "FolderPath": "",
                "FileName": ""
            },
            "MimeType": "image/png",
            "FolderPath": "progression/Inventory/Emblems",
            "FileName": "emblem_palette.PNG"
        },
        "Quality": "Common",
        "ManufacturerId": 0,
        "Type": "None",
        "RewardTrack": "",
        "ParentPaths": [],
        "SortingMetadata": {
            "categoryWeight": 0,
            "subCategoryWeight": 0
        },
        "SeasonNumber": 1,
        "OriginalSeasonNumber": 1,
        "IsCrossCompatible": false,
        "Season": {
            "status": "Frozen",
            "value": "",
            "translations": {}
        }
    }
}

jackphillips31 commented 7 months ago

@jackphillips31 @acurtis166 Actually while I was writing that post I found a kind of hacky way to do it. I wanted to post it anyways just to answer it myself lol.

Solution

I found this solution while looking at the Halo Waypoint website trying to figure out how they're getting their nameplate image. If you copy the image URL though it'll give you a link like this: blob:https://www.halowaypoint.com/8144fcc0-ea16-4db0-a68c-03b3b96c888d and this link ONLY works while you have that session open.

I wasn't hopeful when inspecting the network traffic because I assumed all the requests would be done server-side. But lo and behold... there it was! Halo Waypoint network inspecting nameplate source. This endpoint only requires your Spartan Token. Without further adoe, here are the endpoints...

Endpoint for Emblems: https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/images/emblems Endpoint for nameplates: https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/images/nameplates

But how do we get the filename?

Simple. When you make your Economy_PlayerCustomizationAppearance API call, you should get a response that looks something like this:

{
    "Status": "Success",
    "Appearance": {
        "LastModifiedDateUtc": {
            "ISO8601Date": "2024-03-11T22:17:53.733Z"
        },
        "ActionPosePath": "Inventory/Spartan/ActionPoses/101-000-menu-stance-r-e47e42c4.json",
        "BackdropImagePath": "Inventory/Spartan/BackdropImages/103-000-ui-background-e5fd3381.json",
        "Emblem": {
            "EmblemPath": "Inventory/Spartan/Emblems/104-001-olympus-skull-2845463e.json",
            "ConfigurationId": -2101285248
        },
        "ServiceTag": "MVP",
        "IntroEmotePath": null,
        "PlayerTitlePath": null
    }
}

You're going to want to take note of both the ConfigurationId and the EmblemPath From here you're going to want to use the GameCms_GetItem API call with the EmblemPath. The response you get from that is a little too big to post so I'll cut out what isn't important but it should look like this:

{
    "TagId": 0,
    "ThemeName": {
        "m_identifier": 626314953
    },
    "EmblemShaderName": {
        "m_identifier": -1
    },
    ...
    "CommonData": {
        ...
        "AltName": {
            "status": "Test",
            "value": "104-001-olympus-skull-2845463e",
            "translations": {
                "qps-ploc": "104-001-òŀумφúš-śκΰłŀ-2845463é !!! !!! !!!",
                "qps-ploca": "104-001-ŏŀýmφųѕ-ŝкūłℓ-2845463é !!! !!! !!!",
                "qps-plocm": "104-001-őļўmρΰѕ-ŝķũŀł-2845463з !!! !!! !!!"
            }
        },
        ...
    }
}

What is important here is the response.CommonData.AltName.value in this case being 104-001-olympus-skull-2845463e. Bare with me, we are almost there. This is where the ConfigurationId comes into play. You're going to want to slap on the ConfigurationId onto the back of the AltName string.

If your ConfigurationId is negative then replace the - sign with an n. In my example the ConfigurationId is -2101285248 so we would be adding n2101285248 to the end of the AltName string. Mind you there will be an underscore inbetween and of course we will also be adding a .png to the end as well.

So our finished filename will look like this: 104-001-olympus-skull-2845463e_n2101285248.png Using the endpoints above along with our Spartan Token, we can now retrieve the colored versions of both our emblems and nameplates.

Emblem URL: https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/images/emblems/104-001-olympus-skull-2845463e_n2101285248.png Nameplate URL: https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/images/nameplates/104-001-olympus-skull-2845463e_n2101285248.png

Thank you for reading and if this is new, please enjoy!

acurtis166 commented 7 months ago

@jackphillips31 Nice! I haven't added any of the economy endpoints so this will be helpful when I get to that. Thanks for sharing!

jackphillips31 commented 7 months ago

@acurtis166 I actually just found an endpoint that gives the paths to all the emblems/nameplates and the text color for their corresponding configuration! The endpoint: https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/images/emblems/mapping.json

The response has MANY more items than this but it looks like:

{
   "104-001-olympus-angry-d0fe80dc":{
      "-1408018525":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_n1408018525.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_n1408018525.png",
         "textColor":"#000000"
      },
      "-711804461":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_n711804461.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_n711804461.png",
         "textColor":"#FFFFFF"
      },
      "-1490538315":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_n1490538315.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_n1490538315.png",
         "textColor":"#000000"
      },
      "-1561561067":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_n1561561067.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_n1561561067.png",
         "textColor":"#FFFFFF"
      },
      "1740959434":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_1740959434.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_1740959434.png",
         "textColor":"#000000"
      },
      "1387731827":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_1387731827.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_1387731827.png",
         "textColor":"#FFFFFF"
      },
      "-1788399426":{
         "emblemCmsPath":"images/emblems/104-001-olympus-angry-d0fe80dc_n1788399426.png",
         "nameplateCmsPath":"images/nameplates/104-001-olympus-angry-d0fe80dc_n1788399426.png",
         "textColor":"#000000"
      }
   },
}

dend commented 7 months ago

@acurtis166 @jackphillips31 you might also find this relevant: https://den.dev/blog/openspartan-battlepass/#a-horse-with-no-nameplate

I've been tackling the issue of "missing" nameplates and player emblems, so you might want to familiarize yourself with the exceptions 😊 (the ranked banners are my favorite, but are not available in the API).

aapokaapo commented 7 months ago

Hey guys have you seen this tool: https://github.com/Gamergotten/Infinite-runtime-tagviewer I wonder if this could be used to crack the theater recordings open

dend commented 7 months ago

@aapokaapo haven't looked at it yet, but these past few evenings I've been digging through the film files anyway 😊

ChaseWoodhams commented 6 months ago

@aapokaapo I sent the bin files over to a few awesome Modders I know that are very familiar with this tool. I will report back if anyone cracks the code. I want heatmaps something terrible!

acurtis166 commented 2 months ago

@dend @aapokaapo @ChaseWoodhams Not sure if you guys have had any updates on the film data, but I wanted to share some of my results from looking at the highlight events chunk type files in case any of you are able to do something fun with it.

So the highlight event chunk is the last of the chunks referenced by the metadata response and contains the events referenced in the timeline when viewing a match in theater mode.

theater

I was able to parse out these events by finding 64-bit XUIDs preceding specific bytes and then interpreting the binary content that followed. JSON representation of a parsed event:

{
    "xuid": 1234567890123456,
    "gamertag": "Gamer123",
    "type_hint": 50,
    "is_medal": true,
    "event_type": "medal",
    "time_ms": 341426,
    "medal_value": 71,
    "medal_name": "Rifleman"
}

Since then I've been playing with how to use the data and have produced a couple interesting plots.

Plot displaying players' KD-difference over the duration of the match:

kd_time_series

Plot showing killer-victim counts (involves a join between kill events and death events using a 5 ms tolerance):

killer_victim_counts

Unfortunately, I haven't found any indication of player position or weapon usage, though I'm not expecting to given the events seem to just be used for navigation. I'd still like to see if that information can be pulled from the replication data chunks.

If you are interested in the code, it can be found here.

Here's a minimal example using spnkr.

import asyncio
import json

from aiohttp import ClientSession
from spnkr import HaloInfiniteClient, film

async def main():
    async with ClientSession() as session:
        client = HaloInfiniteClient(
            session, spartan_token="SPARTAN_TOKEN", clearance_token=""
        )
        events = await film.read_highlight_events(client, match_id="MATCH_GUID")

    with open("highlight_events.json", "w") as fp:
        json.dump([e._asdict() for e in events], fp)

if __name__ == "__main__":
    asyncio.run(main())

aapokaapo commented 2 months ago

Amazing work!

I had pretty much given up with this, since cracking the code was way out of my skills.

dend commented 2 months ago

Awesome job @acurtis166. The more I dig through it the more I am convinced that the positioning data is for the engine rather some kind of direct coordinates on the map. I am trying to replicate that with a custom game to see how values change over the same map.

dend commented 2 months ago

@acurtis166 I was doing some tests and seems like the "last chunk" analysis could be significantly flawed. For example, take this match film (one of my Husky Raid games):

https://discovery-infiniteugc.svc.halowaypoint.com
   /hi
   /films
   /matches
   /4fb89c93-53e1-4d7e-b273-5f4c4c1a58e4
   /spectate

Looking at the overall game performance in the PGCR:

Screenshot of the Post-Game Carnage Report in Halo Infinite

Quick stats:

34 kills
11 deaths
24 medals

However, looking at /filmChunk22 alone, specifically for my gamertag, I only see 10 occurrences:

Binary text in a hex editor

While you are right that the last chunk is the chunk that logs all events, we should check if these events are not grouped or otherwise compressed into the player data. Will be digging through it more.

acurtis166 commented 2 months ago

@dend Yes! That held me up for a long time, but then I realized the events aren't byte-aligned.

I'm sure there are plenty of tools for this, but I used the Python package bitstring:

import zlib

import bitstring

file = ...  # Path to compressed highlight events chunk
byte_aligned = ...  # True | False

with open(file, "rb") as fp:
    data = zlib.decompress(fp.read())

bits = bitstring.Bits(bytes=data)
term = "ZeBond".encode("utf-16le")

n = 0
for start in bits.findall(term, bytealigned=byte_aligned):
    remainder = start % 8
    hexstr = bits[start : start + (60 * 8)].tobytes().hex(" ")
    message = f"start={start:<8} | remainder={remainder} | hex={hexstr}"
    print(message)
    n += 1

print(n, "found")

With byte_aligned = True:

start=270496   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 b4 b6 00 00 00 00 00 00 00 00
start=376440   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 dc 00 00 c2 54 00 00 00 01 00 00 00 02       
start=588456   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 00 ce d8 00 00 00 00 00 00 00 01       
start=2931864  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 02 d3 97 00 00 00 00 00 00 00 02       
start=3652600  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 03 55 04 00 00 00 01 00 00 00 00       
start=4820824  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 32 a9 00 00 00 01 00 00 00 4b       
start=4997456  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 04 4e 29 00 00 00 01 00 00 00 63       
start=5224664  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 6a 61 00 00 00 00 00 00 00 00       
start=5648592  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 04 e6 a9 00 00 00 00 00 00 00 01       
start=7194120  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 96 00 05 f1 c8 00 00 00 01 00 00 00 20       
10 found

With byte_aligned = False:

start=23204    | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 9e 2a 00 00 00 00 00 00 00 00
start=176269   | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 a8 86 00 00 00 01 00 00 00 83       
start=199825   | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 a8 86 00 00 00 00 00 00 00 00       
start=246940   | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 00 a8 87 00 00 00 01 00 00 00 00       
start=270496   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 b4 b6 00 00 00 00 00 00 00 00       
start=306210   | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 96 00 00 b4 b6 00 00 00 01 00 00 00 01       
start=329766   | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 c2 54 00 00 00 00 00 00 00 00       
start=376440   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 dc 00 00 c2 54 00 00 00 01 00 00 00 02       
start=447111   | remainder=7 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 cc 0b 00 00 00 00 00 00 00 00       
start=494226   | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 00 cc 0b 00 00 00 01 00 00 00 09       
start=517782   | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 e1 00 00 cc 0b 00 00 00 01 00 00 00 03       
start=588456   | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 00 ce d8 00 00 00 00 00 00 00 01       
start=788198   | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 04 8c 00 00 00 00 00 00 00 00       
start=941698   | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 2c b0 00 00 00 00 00 00 00 00       
start=988372   | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 2d 15 00 00 00 00 00 00 00 00       
start=1024086  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 01 2d 15 00 00 00 01 00 00 00 00       
start=1047642  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 3c 54 00 00 00 00 00 00 00 00       
start=1094757  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 96 00 01 3c 55 00 00 00 01 00 00 00 01       
start=1164549  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 54 92 00 00 00 00 00 00 00 00       
start=1200263  | remainder=7 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 01 54 92 00 00 00 01 00 00 00 09       
start=1317611  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 6b 0e 00 00 00 00 00 00 00 00       
start=1364726  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 6b 0e 00 00 00 01 00 00 00 4c       
start=1388282  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 82 c7 00 00 00 00 00 00 00 00       
start=1612778  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 01 af 4d 00 00 00 00 00 00 00 00       
start=1730883  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 01 cb 74 00 00 00 00 00 00 00 02       
start=1801557  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 01 d5 09 00 00 00 00 00 00 00 01       
start=2048859  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 02 0f 7d 00 00 00 01 00 00 00 64       
start=2072415  | remainder=7 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 02 0f 7e 00 00 00 00 00 00 00 00       
start=2119530  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 02 25 21 00 00 00 00 00 00 00 00       
start=2178805  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 02 27 05 00 00 00 00 00 00 00 01       
start=2565699  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 02 78 39 00 00 00 00 00 00 00 01       
start=2625729  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 02 9c 64 00 00 00 00 00 00 00 00       
start=2814513  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 02 cf 7b 00 00 00 00 00 00 00 00       
start=2931864  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 02 d3 97 00 00 00 00 00 00 00 02       
start=2978981  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 02 d8 bd 00 00 00 00 00 00 00 01       
start=3214131  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 0a 34 00 00 00 00 00 00 00 00
start=3333439  | remainder=7 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 03 0d fc 00 00 00 00 00 00 00 01       
start=3369153  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 17 d1 00 00 00 01 00 00 00 5b       
start=3534814  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 42 a3 00 00 00 00 00 00 00 00       
start=3581929  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 42 a3 00 00 00 01 00 00 00 48       
start=3605485  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 55 04 00 00 00 00 00 00 00 00       
start=3652600  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 03 55 04 00 00 00 01 00 00 00 00       
start=3699717  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 03 5c 1f 00 00 00 00 00 00 00 01       
start=3899905  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 03 80 39 00 00 00 00 00 00 00 01       
start=4242206  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 03 bd 9d 00 00 00 00 00 00 00 02       
start=4265762  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 bf 3d 00 00 00 00 00 00 00 00       
start=4395714  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 03 de 75 00 00 00 00 00 00 00 00       
start=4513819  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 07 c6 00 00 00 00 00 00 00 00       
start=4620212  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 1a 28 00 00 00 00 00 00 00 00       
start=4655926  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 04 1a 28 00 00 00 01 00 00 00 00       
start=4679482  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 21 b7 00 00 00 00 00 00 00 00       
start=4726597  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 04 21 b7 00 00 00 01 00 00 00 09       
start=4750153  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 96 00 04 21 b7 00 00 00 01 00 00 00 01       
start=4773709  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 32 a9 00 00 00 00 00 00 00 00       
start=4820824  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 32 a9 00 00 00 01 00 00 00 4b       
start=4844380  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 dc 00 04 32 a9 00 00 00 01 00 00 00 02       
start=4997456  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 04 4e 29 00 00 00 01 00 00 00 63
start=5021012  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 4e 29 00 00 00 00 00 00 00 00       
start=5224664  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 6a 61 00 00 00 00 00 00 00 00       
start=5318899  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 04 76 7f 00 00 00 00 00 00 00 01       
start=5389132  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 04 a8 07 00 00 00 00 00 00 00 00       
start=5507681  | remainder=1 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 04 bf 4d 00 00 00 00 00 00 00 01       
start=5648592  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 14 00 04 e6 a9 00 00 00 00 00 00 00 01       
start=5920970  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 11 26 00 00 00 00 00 00 00 00       
start=6003358  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 05 16 1b 00 00 00 00 00 00 00 02       
start=6026914  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 33 4c 00 00 00 00 00 00 00 00       
start=6238063  | remainder=7 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 49 31 00 00 00 01 00 00 00 69       
start=6261619  | remainder=3 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 49 31 00 00 00 00 00 00 00 00       
start=6308734  | remainder=6 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 05 54 25 00 00 00 00 00 00 00 02       
start=6462996  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 05 7c d0 00 00 00 00 00 00 00 02       
start=6747962  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 a7 6d 00 00 00 00 00 00 00 00       
start=6934669  | remainder=5 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 32 00 05 c5 fe 00 00 00 00 00 00 00 00       
start=6981786  | remainder=2 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 64 00 05 c5 fe 00 00 00 01 00 00 00 09       
start=7170564  | remainder=4 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 05 f1 ba 00 00 00 00 00 00 00 02       
start=7194120  | remainder=0 | hex=5a 00 65 00 42 00 6f 00 6e 00 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01 01 00 00 00 00 00 00 00 00 00 00 00 00 96 00 05 f1 c8 00 00 00 01 00 00 00 20       
75 found

The 60 bytes dumped above line up with the tokens here. Definitely not claiming to have the "correct" parsing there, but I've checked a few thousand matches with high accuracy in terms of kill, death, and medal counts per player. One gotcha is that AI kills are not recorded (firefight, minigame, etc.).

I also dumped the highlight events to JSON to double check:

highlight_events.json

dend commented 2 months ago

@acurtis166 ah, good catch - I had to fiddle a bit with my C# logic there, but this is what it would look like:

namespace ComponentSearchByteAlign
{
    internal class Program
    {
        public static void Main(string[] args)
        {
            byte[] data = File.ReadAllBytes(@"X:\drive\decompressed_output22.bin");
            byte[] pattern = { 0x5A, 0x00, 0x65, 0x00, 0x42, 0x00, 0x6F, 0x00, 0x6E, 0x00, 0x64 };

            List<int> matchPositions = FindPattern(data, pattern);

            if (matchPositions.Count > 0)
            {
                Console.WriteLine("Pattern found at bit positions:");
                foreach (int position in matchPositions)
                {
                    Console.WriteLine(position);
                }
            }
            else
            {
                Console.WriteLine("Pattern not found.");
            }
        }

        public static List<int> FindPattern(byte[] data, byte[] pattern)
        {
            List<int> matchPositions = [];
            int dataBitLength = data.Length * 8;
            int patternBitLength = pattern.Length * 8;

            for (int bitPos = 0; bitPos <= dataBitLength - patternBitLength; bitPos++)
            {
                if (IsBitMatch(data, pattern, bitPos))
                {
                    matchPositions.Add(bitPos);
                }
            }
            return matchPositions;
        }

        public static bool IsBitMatch(byte[] data, byte[] pattern, int bitOffset)
        {
            // Calculates the number of whole bytes to skip.
            // We divide bitOffset by 8 because there are 8 bits per byte.
            int byteOffset = bitOffset / 8;

            // Calculates how far into the byte (number of bits) we need to start.
            // It's the remainder when bitOffset is divided by 8, giving the bit position within the byte.
            int bitShift = bitOffset % 8;

            // On the above, a good example to visualize the behavior:
            // If bitOffset = 10, byteOffset = 1 (skip 1 full byte) and bitShift = 2 (start at the 3rd bit in the second byte - we skip 2).

            // We now iterate through every byte in the pattern that is given to
            // us when the function is called.
            for (int i = 0; i < pattern.Length; i++)
            {
                // Get the data byte that alligns with the current
                // pattern byte and shifts the bits to the left by the
                // calculated bit shift value earlier.
                byte dataByte = (byte)(data[byteOffset + i] << bitShift);

                // If bitShift > 0, include bits from the next byte. This is
                // important for scenarios where, for example, we're shifting
                // by 3 bits, meaning that part of the data will come from the
                // next byte.
                if (byteOffset + i + 1 < data.Length && bitShift > 0)
                {
                    // Shifts the next byte to the right by the delta between 8
                    // and the calcualted bit shift value, aligning it with the
                    // remaining part of the data byte.
                    // Note: bitwise OR (|=) is used to combine the shifted parts
                    // so that we can perform a full byte comparison.
                    dataByte |= (byte)(data[byteOffset + i + 1] >> (8 - bitShift));
                }

                // Compare dataByte with the current byte in the pattern
                if (dataByte != pattern[i])
                {
                    // Not matching at position. No point in
                    // continuing.
                    return false;
                }
            }

            // All bits match
            return true;
        }
    }
}

The results match what you shared earlier. I was cross-checking earlier to see whether any of the components were Bond-encoded, but it doesn't seem to be the case from the last film chunk. The gamertag/XUID combos are also found in other chunks, and notice that the chunk counts are generally aligned with the separators on the timeline (might be something there).

Either way, this is going to be one good blog post on tracking down the data and parsing it out, really appreciate you working through this with me, @acurtis166!

dend commented 2 months ago

@acurtis166 although in other chunks, if I look by the gamertag, I mostly see two instances at about the same bit position, so I am not sure how that data properly aligns to any of the events in the chunk. It likely is something related to positioning or movement within a given segment.

acurtis166 commented 2 months ago

@dend One thing I noticed in the replication data chunks was that the first byte preceding the first occurrence of each player's XUID appears to increment for the players in sequence:

start	xuid	preceding byte	xuid hex
4019134	2535445291321133	00	2d c7 ef 5b f9 01 09 00
4042784	2535412760927018	01	2a 5b f9 c8 f1 01 09 00
4055076	2533274860220018	02	72 6e 29 04 00 00 09 00
4067336	2535420852629077	03	55 ee 46 ab f3 01 09 00
4079548	2535437773107238	04	26 f0 d0 9b f7 01 09 00
4103134	2535455288437266	05	12 aa cf af fb 01 09 00
4126338	2533274855849476	06	04 be e6 03 00 00 09 00
4150052	2535468858644382	07	9e 97 a8 d8 fe 01 09 00

Which made me wonder if players might be referenced by this index elsewhere?

dend commented 2 months ago

@acurtis166 let me take a closer look at the whole "packet" for player events. I do wonder if there is some kind deterministic header that we can use to properly identify either event references elsewhere or maybe some additional markers.

dend commented 2 months ago

So, some additional observations, and I am purely looking at the hex editor for this (all byte-aligned):

Looking for my own player events, the headers prior to the gamertag are the same.
There are separate entries that start with the XUID. The count of those matches the count of gamertag-tagged entries.
Every event starts with 12 bytes "header", followed by the gamertag.

Here is a binary template for 010 Editor to quickly scan the extracted binary content (assuming the file represents each extracted chunk):

struct HEADER
{
    char bytes[12];
};

struct GAMERTAG
{
    char bytes[32];
};

struct TYPE
{
    char bytes[1];
};

struct TIMESTAMP
{
    char bytes[4];
};

struct BUFF_PADDING
{
    char bytes[15];
};

struct PADDING
{
    char bytes[3];
};

struct MEDAL_MARKER
{
    char bytes[1];
};

local int offset = 0;

HEADER header <bgcolor=0x659157>;
offset += sizeof(HEADER);
FSeek(offset);

GAMERTAG gt <bgcolor=cGreen>;
offset += sizeof(GAMERTAG);
FSeek(offset);

BUFF_PADDING bp <bgcolor=cBlue>;
offset += sizeof(BUFF_PADDING);
FSeek(offset);

TYPE type <bgcolor=cYellow>;
offset += sizeof(TYPE);
FSeek(offset);

TIMESTAMP ts <bgcolor=cRed>;
offset += sizeof(TIMESTAMP);
FSeek(offset);

PADDING padding <bgcolor=cBlue>;
offset += sizeof(PADDING);
FSeek(offset);

MEDAL_MARKER mm <bgcolor=0xF7AF9D>;
offset += sizeof(MEDAL_MARKER);
FSeek(offset);

PADDING padding <bgcolor=cBlue>;
offset += sizeof(PADDING);
FSeek(offset);

MEDAL_MARKER mtype <bgcolor=0xFFC0CB>;
offset += sizeof(MEDAL_MARKER);
FSeek(offset);

Highlighted binary content in a film chunk

@acurtis166 for the medal list, were those extracted manually or are you referring to an API-based index? That is - the indices don't match the medal metadata (https://gamecms-hacs.svc.halowaypoint.com/hi/Waypoint/file/medals/metadata.json) but they are accurate.

aapokaapo commented 2 months ago

I made a quick test site with @acurtis166 SPNKr.py and Dash app. It's very barebones but I am updating it rapidly: https://aapokaapostats.site/stats/ https://github.com/aapokaapo/HaloDashApp

acurtis166 commented 2 months ago

@dend My first thought was to use the medal metadata, as well. When that didn't pan out I started recording them manually. This was mostly by stepping through theater mode. Going to town on bots in custom games yielded a decent amount, haha. To fill in gaps I crawled through match history and compared my missing medal value counts to the corresponding match stats responses.

The mapping from that work is here. I'm still missing some of the harder-to-orchestrate ones.

acurtis166 commented 2 months ago

Looks like a good start @aapokaapo ! The team kill count time series is perfect for visualizing a team slayer match.

dend commented 2 months ago

@acurtis166 - I might be able to help there. I collected a pretty large corpus of my own matches with OpenSpartan Workshop and might just do an aggregation over them, and then map the medal IDs from the film files to individual medal IDs from the match. Once I have that, I can also open a PR and contribute to your list.

dend commented 2 months ago

@acurtis166 have you encountered matches where the events for a player are not captured in the last chunk at all?

BTW, I am pulling the XUIDs/GT combos from filmChunk0 instead of parsing events from the last chunk - it contains the pairings in fairly standard formats.

At its core, the core would look like this:

public static Dictionary<long, string> ProcessFilmBootstrapData(byte[] data, byte[] pattern)
{
    List<int> patternPositions = FindPatternPositions(data, pattern);

    Dictionary<long, string> players = [];

    foreach (int patternPosition in patternPositions)
    {
        int xuidStartPosition = patternPosition - 8 * 8;
        byte[] xuid = ExtractBitsFromPosition(data, xuidStartPosition, 8 * 8);
        var convertedXuid = ConvertBytesToInt64(xuid);

        // We make sure that XUIDs are not some weird values.
        if (convertedXuid > 0)
        {
            int prePatternPosition = xuidStartPosition - 22 * 8;
            var bytePrefixValidated = AreAllBytesZero(data, prePatternPosition, 22 * 8);

            if (bytePrefixValidated)
            {
                byte[] gamertagData = RemoveZeroBytes(ExtractBitsFromPosition(data, prePatternPosition - 32 * 8, 32 * 8));

                players.TryAdd(ConvertBytesToInt64(xuid), ConvertBytesToText(gamertagData));
            }
        }
    }

    return players;
}

The format is like this:

GAMERTAG	Padding	XUID (8 bytes)	Marker 1	Marker 2
[GAMERTAG - Dynamic Length]	0x00 0x00 0x00 ... (22x)	[XUID]	0x2D	0xC0

acurtis166 commented 2 months ago

@dend I have only run into cases (other than missing AI kill events) where a couple kill/death events might be mysteriously missing for a match. Here's one match: 4841da72-c55a-411f-82fe-31bde5d11b97. I could imagine a case where this could happen for a player that only had one or two events and they got dropped, maybe? It would be nice to load historical matches into theater mode.

That's a good call on the header file. My unverified worry with that was that it would only include players that started the match, but it looks like it would simplify things. One thing I ran into with the highlight events was that after certain game updates the distance between gamertag and XUID positions changed. Also - watch "Marker 1". I do not know why, but rarely I found that marker value show up as 0x25, though the header and event files may differ.

dend commented 2 months ago

@acurtis166 I'll give you an example match, can you help me validate whether you see the same behavior?

Match ID: 0421e4e3-0fbc-4df5-a3d3-3b4615f2f1e3

You can search for my own gamertag (ZeBond - 5A 00 65 00 42 00 6F 00 6E 00 64). I don't see anything in the final chunk.

And yes, I have not checked yet whether the players in the film initialization chunk represent the players that were present at the end of the game, although I hope that the film is fully processed when the match is over. I'll see if I can validate that hypothesis with some of the more recent matches with quitters and joiners 😀

acurtis166 commented 2 months ago

@dend I actually don't think there are supposed to be any events. I'm looking at the match stats response and do not see any kills, deaths, medals, or mode-specific counts for your XUID.

acurtis166 commented 2 months ago

@dend Check this out. Near the end of each replication data chunk there is a 4-byte little-endian integer encoded at the position highlighted below:

If you subtract the equivalent value from the previous chunk, you get values that are suspiciously similar to the duration of the chunk from the film metadata JSON. I've seen similar values at the beginning of and throughout the files as well.

index	timestamp?	difference from last	metadata duration (ms)
01	2294366207	NA	19993
02	2314371049	20004842	20004
03	2334377860	20006811	20006
04	2354387014	20009154	20008
05	2374390076	20003062	20002
06	2394403283	20013207	20012
07	2414408528	20005245	20005
08	2434411157	20002629	20002
09	2454414409	20003252	20003
10	2474417759	20003350	20003
11	2494421731	20003972	20003
12	2514425010	20003279	20003
13	2534437543	20012533	20012
14	2554442309	20004766	20004
15	2574445700	20003391	20003
16	2594449862	20004162	20004
17	2614455029	20005167	20004
18	2634458286	20003257	20002
19	2654464494	20006208	20006
20	2674468757	20004263	20004
21	2694472161	20003404	20003
22	2714478405	20006244	20006
23	2729097740	14619335	14619

acurtis166 commented 2 months ago

@dend Dug a little deeper into my last comment. I haven't figured out a consistent method of extracting all the timestamps across matches, but I was able to do so for a single match using a marker that showed up after each timestamp and decoding the prior 4 bytes to see if the value was within an acceptable range relative to the previously decoded timestamp.

Couple findings:

The "ticks" in the timestamp are greatest in the last third of the file. Below is a plot that shows the timestamp value relative to the starting value for the file. Each line represents a separate replication data chunk.

The distribution of tick lengths has a spike around 16-17ms, which lines up nicely with the server tick rate for arena matches (1000 ms / 60 Hz = 16.7 ms/tick). I will see if BTB comes back with double the length (30 Hz).

The number of bytes within a "tick" seems to increase when there is more activity. I tested in a custom game with a single bot by just getting a couple kills and then checking the byte length of the ticks that bracketed highlight events. The ticks with the kills had byte lengths an order of magnitude higher than the baseline. The plot below shows 5 seconds of the match around one of the kill timestamps.

acurtis166 commented 2 months ago

@dend Big team battle:

dend commented 2 months ago

@dend I actually don't think there are supposed to be any events. I'm looking at the match stats response and do not see any kills, deaths, medals, or mode-specific counts for your XUID.

@acurtis166 - aha, that was one match where I actually left mid-game (I believe it was on a bad map), so that makes sense. But here is another one for you:

Match ID: a60ca05a-c965-439a-b728-46cbdf17a256

Look up this player: 2533274830598241 - BOSS IS TOO PRN (the hex version is 42 00 4F 00 53 00 53 00 20 00 49 00 53 00 20 00 54 00 4F 00 4F 00 20 00 50 00 52 00 4E). Do you see anything?

acurtis166 commented 2 months ago

@dend I do see the 60 events. It looks like maybe the the gamertag you are using might be off. I'm seeing "BOSS IS TOO PRO".

dend commented 2 months ago

Aha! Good catch @acurtis166 - looks like my initial GT extraction logic is flawed when it comes to extraction. Adjusting as we speak.

dend commented 2 months ago

@acurtis166 found the flaw in my logic - needed to update this piece in the extraction:

extractedData[byteCount - 1] &= (byte)(0xFF >> (7 - endBitShift));

dend commented 2 months ago

So, confirmed - what's in filmChunk0 are only players that were there at the start of the match. It does not contain players that were there at the end of the match. End-of-the-match players are captured in the final chunk (type 3).

dend commented 2 months ago

Here is a fun match to analyze: a9a19635-ab78-446c-8b0a-582949c318cc.

Here are the XUIDs and players that we've extracted from the film bootstrap chunk:

XUID	XUID hex	GT	GT hex
2533274855333605	`E5 DE DE 03 00 00 09 00`	ZeBond	`5A 00 65 00 42 00 6F 00 6E 00 64`
2535444125582900	`34 06 74 16 F9 01 09 00`	Brightbuddy	`42 00 72 00 69 00 67 00 68 00 74 00 62 00 75 00 64 00 64 00 79`
2535420913081024	`C0 5A E1 AE F3 01 09 00`	D4RK90HAZE	`44 00 34 00 52 00 4B 00 39 00 30 00 48 00 41 00 5A 00 45`
2535453467537243	`5B EF 46 43 FB 01 09 00`	Kendrothma	`4B 00 65 00 6E 00 64 00 72 00 6F 00 74 00 68 00 6D 00 61`
2535420184563877	`A5 10 75 83 F3 01 09 00`	Killa2urDome	`4B 00 69 00 6C 00 6C 00 61 00 32 00 75 00 72 00 44 00 6F 00 6D 00 65`
2535443540376873	`29 7D 92 F3 F8 01 09 00`	SSJ ANGELITO24	`53 00 53 00 4A 00 20 00 41 00 4E 00 47 00 45 00 4C 00 49 00 54 00 4F 00 32 00 34`
2535414687920931	`23 F3 D4 3B F2 01 09 00`	DrewStylez	`44 00 72 00 65 00 77 00 53 00 74 00 79 00 6C 00 65 00 7A`
2535453506777021	`BD AF 9D 45 FB 01 09 00`	psilocybe007	`70 00 73 00 69 00 6C 00 6F 00 63 00 79 00 62 00 65 00 30 00 30 00 37`

Now, this data does not represent the players that are actually captured by the match stats endpoint:

https://halostats.svc.halowaypoint.com/hi/matches/a9a19635-ab78-446c-8b0a-582949c318cc/stats

Which returns a few more XUIDs that we haven't seen above (the table is complete, including bid bots):

XUID	XUID hex	GT	GT hex
2533274855333605	`E5 DE DE 03 00 00 09 00`	ZeBond	`5A 00 65 00 42 00 6F 00 6E 00 64`
2535444125582900	`34 06 74 16 F9 01 09 00`	Brightbuddy	`42 00 72 00 69 00 67 00 68 00 74 00 62 00 75 00 64 00 64 00 79`
2535420913081024	`C0 5A E1 AE F3 01 09 00`	D4RK90HAZE	`44 00 34 00 52 00 4B 00 39 00 30 00 48 00 41 00 5A 00 45`
2535453467537243	`5B EF 46 43 FB 01 09 00`	Kendrothma	`4B 00 65 00 6E 00 64 00 72 00 6F 00 74 00 68 00 6D 00 61`
2535420184563877	`A5 10 75 83 F3 01 09 00`	Killa2urDome	`4B 00 69 00 6C 00 6C 00 61 00 32 00 75 00 72 00 44 00 6F 00 6D 00 65`
2535443540376873	`29 7D 92 F3 F8 01 09 00`	SSJ ANGELITO24	`53 00 53 00 4A 00 20 00 41 00 4E 00 47 00 45 00 4C 00 49 00 54 00 4F 00 32 00 34`
2535414687920931	`23 F3 D4 3B F2 01 09 00`	DrewStylez	`44 00 72 00 65 00 77 00 53 00 74 00 79 00 6C 00 65 00 7A`
2535453506777021	`BD AF 9D 45 FB 01 09 00`	psilocybe007	`70 00 73 00 69 00 6C 00 6F 00 63 00 79 00 62 00 65 00 30 00 30 00 37`
bid(41.0)	N/A	N/A	N/A
bid(57.0)	N/A	N/A	N/A
bid(0.0)	N/A	N/A	N/A
2535409714771833	`79 B7 68 13 F1 01 09 00`	DarthChemo31	`44 00 61 00 72 00 74 00 68 00 43 00 68 00 65 00 6D 00 6F 00 33 00 31`
2533274943130338	`E2 8A 1A 09 00 00 09 00`	RestrictedOreo	`52 00 65 00 73 00 74 00 72 00 69 00 63 00 74 00 65 00 64 00 4F 00 72 00 65 00 6F`
2535464010682428	`3C 90 B2 B7 FD 01 09 00`	gamerbb3311	`67 00 61 00 6D 00 65 00 72 00 62 00 62 00 33 00 33 00 31 00 31`

So there are 3 players that were added to the game at the end. This makes it super-convenient to use the "bootstrap" chunk to learn about who was there at the start of the game and then the end chunk to see who was there at the end. Now, you could arguably also do this by looking at the match endpoint metadata and correlate between PresentAtBeginning, LeftInProgress, and PresentAtCompletion, but films make it easy to also correlate those with events.

Now, the next important step was identifying how to reliably extract Gamertag + XUID combos from the "final" chunk, given that the structure is slightly different than in the "bootstrap" chunk.

The pattern for the XUID identification is the same (I am running with 0x2D 0x2C for now, assuming that 0x25 is not frequent). Looking at byte-aligned data, the gap between GT and XUID was different from piece to piece and might represent different things. For example, the first instance of 0x2D 0xC0 had 2535453467537243 (5B EF 46 43 FB 01 09 00) XUID next to it, and the GT before it was psilocybe007, but that's clearly not the right XUID for it (should be 2535453506777021, i.e., BD AF 9D 45 FB 01 09 00). Within the same "chunk" of data, there is also a reference to 29 7D 92 F3 F8 01 09 00, which is 2535443540376873 (SSJ ANGELITO24) - also not the right association. So looks like from the final file we can't really quickly map out GT+XUID from different message chunks, since they are not always associated with each other. However, that same metadata format that I explained earlier - GT+XUID combo in /filmChunk0 is also available in other chunks (not the last one).

dend / blog-comments

blog/halo-api-match-stats/ #5

Getting Halo Infinite Match Stats With Official Halo API · Den Delimarsky

Hey Den!

Example

Endpoint

JSON Response

Questions Regarding Paths

Conclusion

Questions

Rank Images

Player Banners

Thank you for your reply!

Another Question...

I get the idea...

Thank you again!

TLDR: How do I get the player nameplates and the different colored emblems?

Economy_PlayerCustomizationAppearance

GameCms_GetItem

Configurations

Solution

But how do we get the filename?

Thank you for reading and if this is new, please enjoy!