Hello.
It would be helpful for one of my usecases of baron if it supported non ascii identifiers.
Currently (in 0.6.6 but probably also in the latest master) the following code:
>>> baron.parse("fňuk=42")
Raises the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python3\lib\site-packages\baron\baron.py", line 49, in parse
tokens = tokenize(source_code, False)
File "C:\Python3\lib\site-packages\baron\baron.py", line 70, in tokenize
return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))
File "C:\Python3\lib\site-packages\baron\spliter.py", line 6, in split
return list(split_generator(sequence))
File "C:\Python3\lib\site-packages\baron\spliter.py", line 76, in split_generator
raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: 'ňuk=42\n'
The reason looks quite clear, during the splitting the condition block with the identifier chars uses ascii only.
It would be probably easy to fix this error without taking into account any python 2.x compatibility, but if it is desired, then it is more complicated.
But anyway, could something be done with it?
asterbini
commented
4 years ago
Hello. It would be helpful for one of my usecases of baron if it supported non ascii identifiers. Currently (in 0.6.6 but probably also in the latest master) the following code:
>>> baron.parse("fňuk=42")Raises the following:The reason looks quite clear, during the splitting the condition block with the identifier chars uses ascii only. It would be probably easy to fix this error without taking into account any python 2.x compatibility, but if it is desired, then it is more complicated. But anyway, could something be done with it?