Parse Chat Messages to Build Social Graph

[benhoyle]

The idea is we use spaCy to parse chat messages as they are received.

We can extract named entitys and noun chunks from the chat messages. We can then use these to build a graph representation of the user's social environment (friends and family). The graph can then be retrieved and used as a chat context.

Our general graph model has Nodes and Edges. Nodes could have the following types:

• People
• Events
• Places

There will also be a time element. For example, the nodes in the user's life will change over time, e.g. marriage/re-marriage, death, birth.

To do:

• [ ] Create functions that parse a chat message and determine people

[benhoyle]

We want to work on the chat text. But the parsing methods should be relatively independent of the chat creation logic.

We have a ChatReference model in models.chats.

I think the aim was to use this to model each of the three different types of knowledge - event, person, and place.

But it's a bit confusing coming back to it and forgetting what it is meant to be doing.

[benhoyle]

What is the non-confusing way?

• Each of event, person, and place (e.g. nodes) would have a ChatReference record that would indicate where in the text words associated with the event/person/place occurred (character index), token indexes for the spacy doc for the span, a sentence index in the chat

What's the doc index?

Is it the event/person/place that needs the chat reference or the graph Node?

At the moment we use HasChatReferences as a mixin on the individual Event, Person, and Place models.

One issue is we need a many-many relationship between Node and Chat - a node can be linked with many chats and each chat can be linked with many nodes.

• [x] Use a chat reference on the node instead - it's easier to understand

[benhoyle]

A decent write-up of some explorations into graphs

• https://yoheinakajima.com/knowledge-graph-%f0%9f%a4%9d-llms/
• https://github.com/yoheinakajima/prettygraph

[benhoyle]

Parsing Function

So back to the parsing function.

Ideally we want something independent of our data model. So the core logic would be:

• Have text as input
• Parse with spacy to generate doc
• Get the ents from the doc
• Output candidate knowledge graph (nodes + any edges) just using spacy

Then we want to parse with an LLM to complement the ent extraction (e.g., non-named entities).

Then we want to reconcile with existing nodes and edges in the DB.

Generating a Knowledge Graph with Spacy

• We first extract the ents - these form the nodes
• We then process the dependencies between the ents - these form the edges

[benhoyle]

GPT4-turbo is actually pretty good at this at a sentence level:

[benhoyle]

You counter-intuitively get better performance when you ask for the nodes and edges at the same time?

No you get the same variation with nodes and edges...

[benhoyle]

Interestingly - it doesn't get the two "Karens" if we just give it the list of tokens and indexes:

[benhoyle]

But doesn't get all the "he"s etc out of the box:

[benhoyle]

gpt4-o is quite good at this:

[benhoyle]

To do:

• [ ] Generate the nodes with a spaCy parse
• [ ] Generate the notes with an LLM
• [ ] Consolidate the nodes into a single list
• [ ] Pass the nodes into prompts to generate the edges
• [ ] Parse the JSON indicating the edges and label up the tokens again

[benhoyle]

Observations

• You need to label/extract entities first so these become the context for the relationships. The entities need to appear in the text prompt or fed-back output to be available as "memory" to the LLM when generating the relationship information. The LLMs cannot generate entities and relationships inline.