search

Spandan-Madan / Me_Bot

Build a bot that speaks like you!
681 stars 68 forks source link

Any one want to add support for messenger chats? Wechat? Telegram? #5

Open Spandan-Madan opened 5 years ago

Spandan-Madan commented 5 years ago

This is a proof of concept, and help in extending it to other kinds of chat sources would be helpful. If someone can make a PR, I would be more than happy to accept it!

Essentially, you would need to make a clean_messenger_chat.py or clean_wechat_chat.py file to create all the pickle files as are currently created from WhatsApp chats.

Thanks!

kwikiel commented 5 years ago

What's the format of your_sents.p? - it would be interesting to add messenger support but it's unclear for me what's the structure of those intermediary files since I don't have Whatsapp

JafarAkhondali commented 5 years ago

@Spandan-Madan I would like to try creating Telegram version, But it's better to put some comments in your code ( I know it's clean but putting some comments helps other understand the concepts and understand code without reading it)

ghost commented 5 years ago

@Spandan-Madan I'd like to try WeChat version, but I'm wondering would it works if the chat context is in another language (e.g. Chinese)? I guess it's relevent with the pretrained model from tensorflow hub

Spandan-Madan commented 5 years ago

@kwikiel You can take a look at the python file clean_whatsapp_chats.py to see how I'm parsing for more details. But to give you some direction - your_sents.p just stores a list of your sentences as a pickle file.

Broadly, when you parse, you care about 3 things -

1) a list of sentences you said. 2) list of sentences said by the other person. 3) A dictionary mapping their sentences to your corresponding responses.

The prepare_files.ipynb file basically works with these, and embeds yours and their sentences using tensorflow's universal sentence encoder. Does some other stuff too, if you read the code it should be clear but if you have any questions, feel free to ask here and I will respond! Best of luck!

Spandan-Madan commented 5 years ago

@JafarAkhondali @kwikiel I can add comments to make it more understandable. I am very busy with research these days so I might take a few days to get around to it, but will do it asap. In the meantime, try working with it and please ask here if you have any questions, Thanks a lot!

Spandan-Madan commented 5 years ago

@DhDiego For other languages I have an idea that you can try.

You'll have to use Facebook's fasttext instead of google's universal sentence encoder. Fasttext is multilingual and available in over 100 languages. It gives you an embedding for a word. So, you can potentially take the average of the fasttext embeddings of the words as the embedding of the sentence. I have tried it before and it works well.

ghost commented 5 years ago

@Spandan-Madan Yeah I'm also thinking about fasttext which use a simple global average to get sentence embedding. Anyway chat records for Wechat is a little bit complex to get (can't save as txt directly) I'm working on this and will follow up.

Spandan-Madan commented 5 years ago

@DhDiego Sounds good, a wechat extension would be pretty cool!

I have worked with average word2vec, fasttext and glove all before. They all seem to do OK, so I think you should definitely give it a shot!

Spandan-Madan commented 5 years ago

@DhDiego @kwikiel Let me know if you need any help with the code base or if something is unclear. Are you still planning on working on the extensions you had mentioned?

ghost commented 5 years ago

@Spandan-Madan The problem about WeChat is that Tencent has saved all the chat history in a database and encrypted it even I download my own chat history. I'm still working on find a stable solution to decrypt it (I found some way to calculate the password base on my own account information but haven't try it). And there is 2 final this week and I'll be free after this Thursday. I believe I'll focus on get the chat data and do the clear script by then.

DiegoD94 commented 5 years ago

@Spandan-Madan Hi This is DhDiego, I change my account to this. And above account is no longer active. What's more I already decrypt chat history database of WeChat and the script will follow up these days

DiegoD94 commented 5 years ago

@Spandan-Madan I already commit a PullRequest for WeChat Extension (English only) working of constructing Chinese version with fasttext or glove

Spandan-Madan commented 5 years ago

@DH-Diego Accepted the pull request for WeChat!! Looking forward to your pull for chinese!

DiegoD94 commented 5 years ago

@Spandan-Madan That's great, I'm working on my final projects and another final exam on 12.20 I'll work on Chinese version after that. I'll try to introduce this fun project to Chinese community after I finish!

Spandan-Madan commented 5 years ago

@DH-Diego Sure! Best of luck for your exams! Also, what's your email ID? I am working on some more interesting projects as well, if you are interested we can collaborate. I am always looking for more people to collaborate with!

DiegoD94 commented 5 years ago

@Spandan-Madan That sounds great, you can reach me via hd2412@columbia.edu and we can do it together. Sorry for late responde