search

pkgcore / pkgcheck

pkgcore-based QA utility for ebuild repos
https://pkgcore.github.io/pkgcheck
BSD 3-Clause "New" or "Revised" License
35 stars 29 forks source link

pkgcheck should use >tree-sitter-0.20.3 #613

Closed thesamesam closed 1 year ago

thesamesam commented 1 year ago

This is with tree-sitter-0.20.3 with ::gentoo at 9a9621d863c8829b54cdd76124c94f83ddd006c5:

~/g/app-shells/bash $ pkgcheck scan
pkgcheck scan: error: Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/pkgcheck/pipeline.py", line 191, in _run_checks
    if results := sorted(
                  ^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/runners.py", line 71, in run
    yield from check.feed(item)
  File "/usr/lib/python3.12/site-packages/pkgcheck/checks/reserved.py", line 144, in feed
    for used_name, *args, lineno in self._feed(pkg):
  File "/usr/lib/python3.12/site-packages/pkgcheck/checks/reserved.py", line 49, in _feed
    item.node_str(node.child_by_field_name("name")): node.start_point
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/bash/__init__.py", line 123, in node_str
    return self.data[node.start_byte : node.end_byte].decode("utf8")
                     ^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'start_byte'
$ pkgcheck --version
pkgcheck 0.10.26-4-gaa668df -- Wed, 2 Aug 2023 14:45:53 +0200
$ pmaint --version
pkgcore 0.12.23-4-g8d1a209 -- Mon, 21 Aug 2023 21:23:37 +0300
thesamesam commented 1 year ago

I recommend using --jobs=1 or it gets confusing quickly.

With debugging prints added:

~/g/app-shells/bash $ pkgcheck scan --jobs=1 --verbose --debug bash-5.1_p16-r6.ebuild
[...]
node=<Node type=variable_name, start_point=(50, 2), end_point=(50, 19)>
self=<<class 'pkgcore.ebuild.ebuild_src.package'> cpv='app-shells/bash-5.1_p16-r6' @0x7efcbb3d4830>
node=<Node type=variable_name, start_point=(51, 2), end_point=(51, 17)>
self=<<class 'pkgcore.ebuild.ebuild_src.package'> cpv='app-shells/bash-5.1_p16-r6' @0x7efcbb3d4830>
node=None
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/pkgcheck/pipeline.py", line 135, in __next__
    result = self._results.popleft()
             ^^^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.12/pkgcheck", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/scripts/__init__.py", line 48, in main
    run(os.path.basename(sys.argv[0]))
  File "/usr/lib/python3.12/site-packages/pkgcheck/scripts/__init__.py", line 40, in run
    sys.exit(tool())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/snakeoil/cli/tool.py", line 81, in __call__
    ret = self.main()
          ^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/cli.py", line 20, in main
    return super().main()
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/snakeoil/cli/tool.py", line 191, in main
    self.handle_exec_exception(e)
  File "/usr/lib/python3.12/site-packages/snakeoil/cli/tool.py", line 176, in main
    exitstatus = func(self.options, self.out, self.err)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/scripts/pkgcheck_scan.py", line 518, in _scan
    for result in pipe:
  File "/usr/lib/python3.12/site-packages/pkgcheck/pipeline.py", line 156, in __next__
    self._kill_pipe(error=results.strip())
  File "/usr/lib/python3.12/site-packages/pkgcheck/pipeline.py", line 124, in _kill_pipe
    raise base.PkgcheckUserException(error)
pkgcheck.base.PkgcheckUserException: Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/pkgcheck/pipeline.py", line 191, in _run_checks
    if results := sorted(
                  ^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/runners.py", line 71, in run
    yield from check.feed(item)
  File "/usr/lib/python3.12/site-packages/pkgcheck/checks/reserved.py", line 144, in feed
    for used_name, *args, lineno in self._feed(pkg):
  File "/usr/lib/python3.12/site-packages/pkgcheck/checks/reserved.py", line 49, in _feed
    item.node_str(node.child_by_field_name("name")): node.start_point
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkgcheck/bash/__init__.py", line 125, in node_str
    return self.data[node.start_byte : node.end_byte].decode("utf8")
                     ^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'start_byte'
thesamesam commented 1 year ago

Minimal reproducer (used arthur's from the other bug as a template):

from tree_sitter import Language, Parser
from ctypes.util import find_library

syslib = find_library("tree-sitter-bash")
lang = Language(syslib, "bash")
parser = Parser()
parser.set_language(lang)
tree = parser.parse(b"""
for ((a=1; a <= 2 ; a++)) ; do
        :;
done
""")

query = lang.query
var_assign_query = query("(variable_assignment) @assign")

for node, _ in var_assign_query.captures(tree.root_node):
        print(f"{node=}")
        print(node.child_by_field_name("name").start_byte)
arthurzam commented 1 year ago

Just for archive and information purpose, tree-sitter-bash did huge work for us to handle all of Gentoo repository, so we don't have any errors. Waiting for newest release (>0.20.3), bundle it, close relevant issues, and remove version cap from ebuild.

arthurzam commented 1 year ago

@thesamesam here are the results of the 2 runs. I already see some changes, which needs some updates in code (adding strip(), adding sorts to some fields, and such):

ts-bash-0.19.0.json ts-bash-0.20.4.json

Highlights

-{"__class__": "EbuildUnquotedVariable", "category": "dev-php", "package": "PEAR-PEAR", "version": "1.10.12", "lines": [118, 122, 126, 130, 134], "variable": "EPREFIX"}
-{"__class__": "EbuildUnquotedVariable", "category": "dev-php", "package": "PEAR-PEAR", "version": "1.10.13", "lines": [118, 122, 126, 130, 134], "variable": "EPREFIX"}
-{"__class__": "EbuildUnquotedVariable", "category": "sys-devel", "package": "gcc-apple", "version": "4.2.1_p5666-r3", "lines": [331, 333], "variable": "EPREFIX"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "8.13", "lines": [388, 389], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "8.13", "lines": [396, 408, 414, 415, 419, 420, 430], "variable": "ED", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "8.14", "lines": [388, 389], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "8.14", "lines": [396, 408, 414, 415, 419, 420, 430], "variable": "ED", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "9999", "lines": [388, 389], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-staging", "version": "9999", "lines": [396, 408, 414, 415, 419, 420, 430], "variable": "ED", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "8.13", "lines": [348, 349], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "8.13", "lines": [356, 368, 374, 375, 379, 380, 390], "variable": "ED", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "8.14", "lines": [348, 349], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "8.14", "lines": [356, 368, 374, 375, 379, 380, 390], "variable": "ED", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "9999", "lines": [348, 349], "variable": "D", "func": "global scope"}
-{"__class__": "EbuildVariableScope", "category": "app-emulation", "package": "wine-vanilla", "version": "9999", "lines": [356, 368, 374, 375, 379, 380, 390], "variable": "ED", "func": "global scope"}
+{"__class__": "EbuildUnquotedVariable", "category": "dev-libs", "package": "openssl-compat", "version": "1.0.2u-r2", "lines": [135], "variable": "EPREFIX"}
+{"__class__": "EbuildUnquotedVariable", "category": "dev-libs", "package": "openssl", "version": "1.0.2u-r1", "lines": [128], "variable": "EPREFIX"}
+{"__class__": "EbuildUnquotedVariable", "category": "gnustep-base", "package": "gnustep-make", "version": "2.8.0-r1", "lines": [56], "variable": "EPREFIX"}
+{"__class__": "EbuildUnquotedVariable", "category": "gnustep-base", "package": "gnustep-make", "version": "2.9.0", "lines": [58], "variable": "EPREFIX"}
+{"__class__": "EclassUnquotedVariable", "eclass": "cuda", "lines": [84, 93, 100], "variable": "EPREFIX"}
+{"__class__": "EclassUnquotedVariable", "eclass": "java-utils-2", "lines": [580, 927], "variable": "D"}
+{"__class__": "EclassUnquotedVariable", "eclass": "kernel-2", "lines": [880], "variable": "WORKDIR"}
+{"__class__": "EclassUnquotedVariable", "eclass": "perl-module", "lines": [257], "variable": "EPREFIX"}