Better default minisearch tokenizer for Chinese documents

[leyen-me]

Describe the bug

搜索不到内容

Reproduction

中文内容

Expected behavior

System Info

no

Additional context

No response

Validations

• [X] Check if you're on the latest VitePress version.
• [X] Follow our Code of Conduct
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

[brc-dd]

Have you tried doing vitepress build then run vitepress preview?

[leyen-me]

您是否尝试过执行 vitepress 构建然后运行 vitepress 预览？

我尝试构建过，你可以预览我的生产链接，https://web.leyen.me

[brc-dd]

https://github.com/lucaong/minisearch/issues/201#issuecomment-2227591121 -- This comment kind of works, but still needs improvement I guess.

There are also some other people doing this - https://github.com/search?q=vitepress+segmenter+language:JavaScript+OR+language:TypeScript+NOT+is:fork&type=code

Not sure but bm25 parameters might help too - https://github.com/search?q=vitepress+searchOptions+bm25+language:JavaScript+OR+language:TypeScript+NOT+is:fork&type=code (I haven't checked how they work yet.)

[brc-dd]

I'm keeping this open. There should be some defaults here instead of needing Chinese users to manually configure it.

[niansi-z]

I tried it out and found that when there was no title, the problem recurred, not just in Chinese.demo

[brc-dd]

With current logic titles are needed. There should be a h1 per page. There is a PR open to make it more robust in handling such content, will see.