search

berkerpeksag / astor

Python AST read/write
https://pypi.org/project/astor/
BSD 3-Clause "New" or "Revised" License
810 stars 102 forks source link

aster converts triple-quote comment to single-quote comment if \x00 is in comment. #225

Open medined opened 3 months ago

medined commented 3 months ago

I am using ast to parse a python file. Then using astor to print a specific function. When the function has the following docstring

        """
            \x00
        """

it is converted to 

'\n            \x00\n        '

by astor.

This causes the python code to fail pylint because the docstring is no longer correct.

P.S. The \x00 in the python comment is part of a longer comment that discusses binary data.

medined commented 3 months ago

I have verified that if my original comment is \x00 then the triple comment is maintained.

medined commented 3 months ago

The issue lies in the following code in string_expr.py:

    try:
        if eval(fancy) == s and '\r' not in fancy:
            return fancy
    except Exception:
        pass

The "source code string cannot contain null bytes" is swallowed. This took me about 4 hours to find. Please send the exception to an error log.

medined commented 3 months ago

I recommend adding the following immediately before the try statement. It's not ideal to change incoming text, but the alternative is that aster is not reliable. This whole issue arose because the OpenDevin project references binary characters in at least one docstring.

s = s.replace('\x00', '\\x00')