Added Chinese and Korean examples to TextTokenizerTest

[Jauntbox]

Related issues n/a

Describe the proposed solution n/a

Describe alternatives you've considered n/a

Additional context This is a small change to better allow testing of alternatives to the CJK tokenizer (that we've already replaced for Japanese). The CJK tokenizer uses bigrams for its tokenization, rather than trying to extract words, so most of the tokens from a text sample will have length 2 (not all, since other languages can be mixed in). Some of the simpler ID detection calculations will look at the distributions of token lengths, so they may incorrectly think that text from languages using the CJK tokenizer is IDs.

[codecov[bot]]

Codecov Report

Merging #442 into master will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #442   +/-   ##
=======================================
  Coverage   86.93%   86.93%           
=======================================
  Files         337      337           
  Lines       11096    11096           
  Branches      362      362           
=======================================
  Hits         9646     9646           
  Misses       1450     1450

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e45073d...9f67367. Read the comment docs.

[tovbinm]

LOCO test is failing @sanmitra @Jauntbox

[sanmitra]

@tovbinm The LOCO test - com.salesforce.op.stages.impl.insights.RecordInsightsLOCOTest is succeeding. Where exactly you are seeing the failure of LOCO test ?

[tovbinm]

It’s a flaky one. See previous runs.