search

rainwoodman / tinypy

tpy, a pet python forked from tinypy
Other
21 stars 5 forks source link

Incorrect handling of text files. #35

Closed mxlgv closed 3 years ago

mxlgv commented 3 years ago

TinyPy crashes with error if the file was created in notepad.exe or some other test editors. Most likely related to '\ n' and '\ r'.

test.txt

File "tinypy/compiler/py2bc.py", line 7, in compile tokens = tokenize.tokenize(s) File "tinypy/compiler/tokenize.py", line 56, in tokenize u_error('tokenize',s,T.f) File "tinypy/compiler/tokenize.py", line 25, in u_error raise Exception('error: '+ctx+'\n'+r)

rainwoodman commented 3 years ago

I'd prefer to run the script via dos2unix rather than add the logic to the compiler to parse both styles of line-ends.

That being said it appears that we already have a transform stage in tokenize.py::clean where we replace \r\n to \n.

rainwoodman commented 3 years ago

It's a bug in tpy_str_replace.

rainwoodman commented 3 years ago

@tubocat2001: Could you confirm if PR #37 fixes this?

mxlgv commented 3 years ago

@rainwoodman, You really fixed that! I am glad!