How to extract semantic labels?

[benjamintanweihao]

I can get object_id, but have no idea how to get the semantic labels.

[bradyz]

Check out https://github.com/philkr/gamehook_gtav/blob/master/gtastate.cpp#L187 I believe this should convert the instance labels to class labels.

Also future issues on the GTA plugin should go in the other repo!

[benjamintanweihao]

Thanks! Thought I thought this might have been not specific to GTA.

I found that doing localhost:8766/raw?t=object_id seems to do what I want. https://github.com/philkr/gamehook/blob/master/plugins/server/server.cpp#L330 points to the same type() in the GTA hook I assume?

Otherwise, how can I make use of the line that you've pointed and use that function?

[bradyz]

So the way it is set up, gamehook simply provides an access point and injects any plugin into a DirectX game and provide raw data from the game. It is then up to the plugin to make sense of the buffers provided (like class labels etc).

If I were you, I would simply port this function to python and run this for every pixel position in object_id buffer and that would give you your per-pixel class labels.

Let me know if that makes sense!

[benjamintanweihao]

AH! Yes, this makes perfect sense. Thank you! 💓

[benjamintanweihao]

As a follow up, here's a small gist to illustrate what @bradyz mentioned: https://gist.github.com/benjamintanweihao/bc935f6e426fd6837852812f93acc979