Unable to parse Arabic text

[jmishra01]

Lark fails to parse Arabic text. Kindly check the sample Python code below to re-generate the issue

import lark

text_1 =  "'راض جداً'"

grammar = """
start: string
string: "'" ( /([\w\s]|''|"|\\'|\\|[`~&!@#$%\^\*\(\)-_=+])+/| /[u"\u0001-\uFFFF"]/ ) "'"

%import common (CNAME, WS, SIGNED_NUMBER, INT)
%import common.NEWLINE -> _NL
%import common.WS_INLINE
%ignore WS_INLINE
%ignore WS
"""

parser = lark.Lark(grammar = grammar)
parse_text_1 = parser.parse(text_1)
print(parse_text_1)

[MegaIng]

Lark is perfectly capable of parsing Arabic text. Your grammar just doesn't match the given text. Depending on what exactly you meant to do, you need to change your definition of string. Most notably, the second term you added | /[u"\u0001-\uFFFF"]/ ) matches (almost) any single character, which probably isn't what you want.

[erezsh]

Your definition of string doesn't include repetition... it can only match a single character.

There are online regexp IDEs, that can help you. You can also test regexps directly using Python's re module.

[jmishra01]

Thanks, @MegaIng and @erezsh, for the quick reply.

The problem is resolved using the below grammar.

grammar = """
start: string
string:  /'([^'\\]*(?:\\.[^'\\]*)*)'/

%import common (CNAME, WS, SIGNED_NUMBER, INT)
%import common.NEWLINE -> _NL
%import common.WS_INLINE
%ignore WS_INLINE
%ignore WS
"""