[FEAT] Character / Domain level information - Storage / Tracking

[dalinarkholin]

🚀 Feature Request

What problem is this feature looking to solve? The Story will kill a character, then in the next scene that character will be alive again, the AI often forgets who is who(pointed out already by @nickwalton). A unique identifier should be applied to a character, and characters should be stored along with their state(alive: boolean) etc.

Describe the solution you'd like There should be some sort of character tracking that tracks characters and a few of their characteristics/attributes as they come out of the AI.

This could be some simple form of storage at first(in memory even - redis?) It would be cool if it was a relational DB that could be added onto as more information comes out about characters(what items they have, etc). The AI would then weight this information a little more heavily in it's responses. As that DB gets larger...I am sure there would be performance considerations.

I am not sure how this feature really translates into a AI Model. But maybe we could train the AI to store important information, instead of explicitly trying to store "characters, etc". If thinking about this from a domain perspective, characters, items, places, etc are all sort of central in this type of game. Not sure what the right route here is. Is there a layer outside of "AI" that is necessary? Or does the AI have to do everything? Maybe there is a layer that feeds the AI background information if it is referenced by the player. I am not sure, just spitting out ideas.

From what I have read so far the model just takes the starting scene and the last 8 entries right? I definitely think some form of long term storage is needed and maybe less understanding of what happened recently, but I guess only if there is a need for a trade off. Both would be great, but obviously response time is already cutting it close on length.

P.S this experience is coming out of the IOS beta release.

[dalinarkholin]

Another thing I noticed is the lack of a realistic skill tree/ any real understanding of what the character can do. For example I pick a Knight, and am able to do anything I want like even shoot fireballs without any training. Learned skills should also be tracked.

[RyushoYosei]

Some tracking would be important, such as if it references the size of something, it should make a record of "This thing is this size." so like, if you meet a dragon that is the size of a housecat, it will remember that dragon is the size of a housecat, so if it grows, it will grow to an actual larger size, For example I have seen things grow from "Dinnerplates" to "Grapefruits" or to basketball sizes, then to grapes. So it seems to need some memory of features, if they are mentioned.

[JWNoctis]

As of now, nothing stops you as a Knight from, say, taking a train to London, then have small talks with the driver after you arrive, and then wade through crowds of delayed passengers out of King's Cross, as long as the model disregarded the fact that there's a live dragon about. Heck, you could even have the dragon waiting for a train on the opposite platform if you so wish...all right I didn't actually try this last one, but I think it's doable.

Personally I'd say this is a feature, and a great, unique feature at that. But I agree that a more consistent experience without manually padding context into actions and copious, cherry-picking reverts would have been nice indeed.

The bigger problem is how to present those unique identifiers and states and object relationships you propose in a form the AI could use. It might be useful to condence the context and previous actions and results into something unequivocal like, say, the dragon is dead, the dragon is killed by you, the damsel is dead too, the damsel was cursed to live as long as the dragon did etc. But I understand that summarisation is a much harder problem in NLP, and getting data like that in the first place - and then doing anything logically consistent with it - might need something more than just GPT-2, or we wouldn't be having this problem.

[RyushoYosei]

As of now, nothing stops you as a Knight from, say, taking a train to London, then have small talks with the driver after you arrive, and then wade through crowds of delayed passengers out of King's Cross, as long as the model disregarded the fact that there's a live dragon about. Heck, you could even have the dragon waiting for a train on the opposite platform if you so wish...all right I didn't actually try this last one, but I think it's doable.

Personally I'd say this is a feature, and a great, unique feature at that. But I agree that a more consistent experience without manually padding context into actions and copious, cherry-picking reverts would have been nice indeed.

The bigger problem is how to present those unique identifiers and states and object relationships you propose in a form the AI could use. It might be useful to condence the context and previous actions and results into something unequivocal like, say, the dragon is dead, the dragon is killed by you, the damsel is dead too, the damsel was cursed to live as long as the dragon did etc. But I understand that summarisation is a much harder problem in NLP, and getting data like that in the first place - and then doing anything logically consistent with it - might need something more than just GPT-2, or we wouldn't be having this problem.

I think you slightl ymissed what I meant, I don 't want the AI to go "No, there is a dragon." Or something like that, I do think it should remember that "I did mention a dragon" and if it mentions anything about that dragon like, it's color, or if it's large, small, male, female, if it has multiple heads.

Basically that it just remembers details about characters that exist, not that it guides you.

[JWNoctis]

I think you slightl ymissed what I meant, I don 't want the AI to go "No, there is a dragon." Or something like that, I do think it should remember that "I did mention a dragon" and if it mentions anything about that dragon like, it's color, or if it's large, small, male, female, if it has multiple heads.

Basically that it just remembers details about characters that exist, not that it guides you.

Trouble is, there really isn't much difference between, for instance, "The dragon has green scales", versus "The dragon is here in front of you, grinning like a Cheshire Cat, as the cave around you glittered with bits of the dragon's hoard, lit by your torch".

And as a matter of fact, the model already does take into account of what had already been presented within provided prompt, and - without going into details of how it works - is just not consistent enough to always do what could be expected, not in the presence of contradictory or interfering data or unusual grammar, and especially not in the presence of existing incoherency in either previous output or the player's action.

It is also not entirely a problem of memory, as it currently often apparently show a lack of reasonable concept of what was presented to it, like your example of grape being larger than grapefruit - Set up a QA session in a custom start, ask a few common-knowledge questions, and see what happens.

Incidentially, the model had placed, of all places, Teignmouth on the northern tip of the isle of Skye in the same adventure as above. To be fair, Teignmouth -was- on one of the main lines to London, but not to King's Cross, and the model had put it on the wrong end of the right Isles.

[Akababa]

This is a really exciting idea. I agree with that it would be difficult to get GPT2 to use structured data. I think the major problem is the training loss only wants to predict the next token, and thus doesn't penalize loss of semantic coherence over multiple tokens. We could try using a multiple-token loss, maybe in combination with a discriminator model that detects incoherence. That could require a lot more training data and compute though.

[kik4444]

This is a really exciting idea. I agree with that it would be difficult to get GPT2 to use structured data. I think the major problem is the training loss only wants to predict the next token, and thus doesn't penalize loss of semantic coherence over multiple tokens. We could try using a multiple-token loss, maybe in combination with a discriminator model that detects incoherence. That could require a lot more training data and compute though.

Can't wait till we have to use 32gb vram gpus lol

[DonaldTsang]

What bout using a Knowledge Graph or something similar as the "base"?