search

google-research / bert

TensorFlow code and pre-trained models for BERT
https://arxiv.org/abs/1810.04805
Apache License 2.0
38.04k stars 9.59k forks source link

Intent prediction from conversation history #421

Open SRHyd opened 5 years ago

SRHyd commented 5 years ago

I want to use BERT for intent classification from text message by considering the conversion history of n messages in a chat session. In addition, there are other features X, Y, Z that we need to consider while predicting the intent.

Let's take an example... Two persons A and B are in the chat session. An and Bn are the messages from person A and B in conversation history up to n depth. A0 and B0 are the last messages from person A and B. The conversation history is represented as AnBn, A(n-1)B(n-1) ....... A1B1, A0B0 My goal here is to classify the intent of the message B0 by considering messages from conversation history up to n depth and X, Y, Z features

My approach: I will modify run_classifier.py to append all messages into one passage and predict [CLS] feature vector from BERT pre-trained model and then, I will concatenate [CLS];X;Y;Z and multiply with weights to get the intent. Is it the correct approach? Please comment...

Thanks SRHyd

Razzaghnoori commented 5 years ago

I believe a serious constraint of your algorithm could be the maximum sequence length (currently 128) of the model which as you may imagine, could be quite irritating. I can think of two other algorithms. But, I also want to suggest a little idea for your own algorithm: remember to make use of token_type_ids. For instance, you can assign 0 to A's messages and 1 to B's.

  1. You can first feed each message to the model and classify the output (extract the intent ignoring other messages). Then train another classifier to accept the sequence of intents and return the intent of the last message. That could be an RNN variant, a transformer or even a weighted majority vote of the previous intents (as it's quite intuitive to care about the most recent messages more).

  2. Another useful tool I believe you can use is the next sentence classifier. BERT is trained to understand if a sentence is the next sentence of the other. So, why not use it? Concatenate sentences that are in the same topic and obtain a semi-dialogue concept and then feed each semi-dialogue to the network. This way you not only avoid dealing with long sequences but you also show the model what is probably a single topic (and has probably the most information you require) at once.

You can, of course, combine these two.

QuangTQV commented 6 months ago

I want to use BERT for intent classification from text message by considering the conversion history of n messages in a chat session. In addition, there are other features X, Y, Z that we need to consider while predicting the intent.

Let's take an example... Two persons A and B are in the chat session. An and Bn are the messages from person A and B in conversation history up to n depth. A0 and B0 are the last messages from person A and B. The conversation history is represented as AnBn, A(n-1)B(n-1) ....... A1B1, A0B0 My goal here is to classify the intent of the message B0 by considering messages from conversation history up to n depth and X, Y, Z features

My approach: I will modify run_classifier.py to append all messages into one passage and predict [CLS] feature vector from BERT pre-trained model and then, I will concatenate [CLS];X;Y;Z and multiply with weights to get the intent. Is it the correct approach? Please comment...

Thanks SRHyd

Have you found an effective solution to this problem yet?