Add Chinese, Japanese, Korean Name support

[LightWind1]

376

I add three regular expression to match Chinese, Japanese, Korean words . Now it can tokenize sql correctly like 'select T2.名称 , T2.南北区域 from 民风彪悍十大城市 as T1 join 省份 as T2 on 民风彪悍十大城市.所属省份id == 省份.词条id group by T1.所属省份id order by count ( * ) asc limit 3'

[andialbrecht]

Hi @LightWind1, can you clarify what problem your change solves? I've had a look on how the parser sees your statement and to me everything looks as expected:

import sqlparse
sql = 'select T2.名称 , T2.南北区域 from 民风彪悍十大城市 as T1 join 省份 as T2 on 民风彪悍十大城市.所属省份id == 省份.词条id group by T1.所属省份id order by count ( * ) asc limit 3'
p = sqlparse.parse(sql)[0]
p._pprint_tree()
|- 0 DML 'select'
|- 1 Whitespace ' '
|- 2 IdentifierList 'T2.名称 ...'
|  |- 0 Identifier 'T2.名称'
|  |  |- 0 Name 'T2'
|  |  |- 1 Punctuation '.'
|  |  `- 2 Name '名称'
|  |- 1 Whitespace ' '
|  |- 2 Punctuation ','
.....and so on.....