search

Siddhant-K-code / OpenAI-bring-your-own-data

Langchain | OpenAI | Train with Custom markdown data
4 stars 2 forks source link

jieba question #1

Open kas84 opened 1 year ago

kas84 commented 1 year ago

Why are you using jieba here? That shouldn't be needed unless you're using Chinese text, right?

Siddhant-K-code commented 1 year ago

Yeah, I still need to do some cleanups & optimizations. But, this was purely just to parse through some specific research papers/txt