Closed sound118 closed 5 years ago
I have no experience with chinese. Can you tell me what happens if you do
a <- test_sentences$text2[9]
predict(model, a)
I have no experience with chinese. Can you tell me what happens if you do
a <- test_sentences$text2[9] predict(model, a)
When you do a <- test_sentences$text2[9] predict(model, a)
It will give an output as below [[1]] label262 0.5312194 This actually gives a wrong prediction which has a low probability of 0.53. However, when you paste the value of "a" which is a tokenized Chinese term "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦" as shown below:
predict(model, "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦")
It will a correct predict with high probability of 0.9935 as below:
[[1]] label314 0.9935217
This is so weird since the two R objects ( a<-test_sentences$text2[9] and c("蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦") are basically the same. Why is there such a quite different prediction?
It may be due to encoding issues. Is your source code file encoded in UTF 8 ?
Another test: test_sentences$text2[9] == "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦"
is it true ?
It may be due to encoding issues. Is your source code file encoded in UTF 8 ? Another test:
test_sentences$text2[9] == "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦"
is it true ?
Yes, the source code file is encoded in UTF-8, `test_sentences$text2[9] == "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦" is true. My train dataset is in excel format and encoded in UTF-8, my test dataset is .txt format encoded in UTF-8, this problem is still happening now... So puzzled!
It may be due to encoding issues. Is your source code file encoded in UTF 8 ? Another test:
test_sentences$text2[9] == "蛋白粉 开封 后 两个 月 在 次 食用 味道 发苦"
is it true ? Thanks to your hint. It is really due to the encoding issue. Once I used predict(model, enc2native(test_sentences[["text2"]])) , I am able to predict all correct labels. Just need to add "enc2native" function for encoding~
Tks for your report, I close the issue as it is not related to the package.
I got an issue with Chinese text classification prediction model as folloing:
Basically, after you trained the model using "fastrtext", if you try to predict a Chinese tokenized text and put it as an object (e.g. test_sentences$text2[9] in my case), it will give you a wrong prediction with low probability. If you just simply copy the tokenized Chinese text into the prediction model like I did above, it will give a correct one with high probability. I am really confused about this situation. Anyone can help with it? Much appreciated!