Account for problems being fixed in Python 3.12

[alexmojaki]

Closes #109

Seems that several things have significantly improved in 3.12!

[alexmojaki]

GHA no longer supports Python 2.7: https://github.com/actions/setup-python/issues/672

While there are ways we could continue testing 2.7, this seems like a good time to let it go. Anyone using 2.7 can still use an older version of asttokens (i.e. the current one) and they should be fine. Newer versions of asttokens have generally been motivated by newer versions of Python anyway, as with this PR.

[hroncok]

Using astroid from git HEAD and this PR, we get this in Fedora:

=================================== FAILURES ===================================
__________________________ TestAstroid.test_fixture9 ___________________________

self = <astroid.builder.AstroidBuilder object at 0x7f4ba936e720>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\nclass Aaaa(base):', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\nclass Aaaa(base):\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\nclass Aaaa(base):\n', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           class Aaaa(base):
E                            ^
E       IndentationError: expected an indented block after class definition on line 2

/usr/lib64/python3.12/ast.py:52: IndentationError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_fixture9>

>   def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')

tests/test_mark_tokens.py:181: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:140: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:804: in parse_snippet
    return self.module.parse('_\n' + text).body[1]
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f4ba936e720>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           expected an indented block after class definition on line 2 (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
_________________________ TestAstroid.test_sys_modules _________________________

self = <astroid.builder.AstroidBuilder object at 0x7f4ba81a0b90>
data = '_\ndef create_module(self, spec):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\ndef create_module(self, spec):', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\ndef create_module(self, spec):\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\ndef create_module(self, spec):\n', filename = '<unknown>'
mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           def create_module(self, spec):
E                                         ^
E       IndentationError: expected an indented block after function definition on line 2

/usr/lib64/python3.12/ast.py:52: IndentationError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]

      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break

        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue

        if not filename:
          continue

        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue

        if self.is_astroid_test and (
            # Astroid fails with a syntax error if a type comment is on its own line
            re.search(r'^\s*# type: ', source, re.MULTILINE)
            # Astroid can fail on this file, specifically raising an exception at this line of code:
            #     lambda node: node.name == "NamedTuple" and node.parent.name == "typing"
            # with the error:
            #     AttributeError: 'If' object has no attribute 'name'
            # See https://github.com/gristlabs/asttokens/runs/7602147792
            # I think the code that causes the problem is:
            #     if sys.version_info >= (3, 11):
            #         NamedTuple = typing.NamedTuple
            or filename.endswith("typing_extensions.py")
        ):
          print('Skipping', filename)
          continue

>       self.create_mark_checker(source)

tests/test_mark_tokens.py:673: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:804: in parse_snippet
    return self.module.parse('_\n' + text).body[1]
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f4ba81a0b90>
data = '_\ndef create_module(self, spec):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           expected an indented block after function definition on line 2 (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
----------------------------- Captured stdout call -----------------------------
/usr/lib64/python3.12/importlib/_bootstrap.py
/usr/lib64/python3.12/importlib/_bootstrap_external.py
=========================== short test summary info ============================
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - astroid.exceptions...
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - astroid.excepti...

[hroncok]

Is it possible that this would in fact be a problem in astroid?

[aqeelat]

@alexmojaki This is not a review of the fix you created but of the deprecation of Python 2.7.

Don't forget to remove the python 2 and 2.7 classifiers and the dependency on six.

Although, I think that should be done in a separate PR that gets merged before this one. I can give it a shot if you create an GH issue and assign it to me.

[alexmojaki]

Don't forget to remove the python 2 and 2.7 classifiers

Thanks, this is an important reminder.

and the dependency on six. Although, I think that should be done in a separate PR that gets merged before this one.

I don't really see why continuing to depend on six is a critical problem that should block this PR (although this PR isn't actually ready to merge yet), even without Python 2 support.

I can give it a shot if you create an GH issue and assign it to me.

I think we need to decide to actually drop Python 2 support, and that's up to @dsagal. But I've opened https://github.com/gristlabs/asttokens/issues/111 for that discussion to continue there.