Can not build graph from most of the projects I have tried

[avijit1258]

Hello,

Thanks for maintaining pyan. I have been trying to use pyan to generate caller-callee relationship in trivial graph format (tgf) for my research work. For example, to get static call graph of this (https://github.com/tiangolo/fastapi) repo I have tried the following commands

/home/avijit/Github/updated_pyan/pyan/pyan3 *.py --uses --no-defines --colored --grouped --annotated -f fastapi.txt --uses --no-defines --tgf By executing above command, I am getting empty file with a '#' which seems like glob failed to load any .py file.

/home/avijit/Github/updated_pyan/pyan/pyan3 fastapi/*.py --uses --no-defines --colored --grouped --annotated -f fastapi.txt --uses --no-defines --tgf

When I tried to point to the source files in fastapi folder, I am getting the following error.

I want to execute pyan for different popular python repositories on Github with nested folder structures which pyan fails most of the time. Please let me know if there is any workaround to make pyan working to generate tgf file for python 3 projects. I would be glad to contribute if there is any need.

Thanks Avijit

[Technologicat]

Hi,

For a nested folder structure, Pyan itself has no recurse-into-subdirectories option, but shells do. If you're using bash, shopt -s globstar (see e.g. here), and then, as the filename pattern, use **/*.py instead of just *.py. If needed, use ls **/*.py to check what it would find.

Pyan should work for Python 3, up to and including 3.7 - but there have been some AST changes in Python 3.6 which it hasn't been extensively tested with. AnnAssign is one of those.

(Note in Python 3.8 there are AST changes that require changes to Pyan before it'll work. Various literals now produce ast.Constant instead of ast.Num, ast.Str, and some others; and there's the new ast.NamedExpr for the walrus operator :=.)

As for the error, thanks for reporting! I didn't immediately see a trivial mistake in the code, so this requires some debugging.

Thanks for the link to a repo which makes the analysis fail - this should make it much easier to track down.

Contributions are welcome! I don't have much time for developing Pyan right now, but I'll be glad to answer any questions.

If you would like to try fixing the error yourself, just let me know. The important things to know are:

• Green Tree Snakes, a.k.a. the missing Python AST documentation
• ast.NodeVisitor
• Look first in pyan/analyzer.py, the other modules are basically infrastructure
• There are two completely different things that are called a node in the analyzer source code: an AST node (as in the output of ast.parse, i.e. the input to the analyzer), and a call graph node (pyan.node.Node, for building the analyzer output).

[avijit1258]

Thank you so much for your detailed reply with resources.I am trying to debug analyzer.py. However, looks like there is some issue with entry point due to recent updates. I can not run pyan3 following the instruction in readme. I have tried pip install pyan3. When I try to use pyan3 from command line it says pyan module not found. Is there any way to run pyan3 from the source code now.

[johnyf]

pip install . or pip install -e . from within the repository (after cloning) would work (using #43).

[Daniel-Mock]

@Technologicat I am interested in adding support for pytest fixture calls. Right now, when I run pyan3 on a basic structure involving pytest tests & fixtures, it shows them as modules & function types, but does not show how they are being called:

Here's how the example fixtures are being called:

I am working my way through Analyzer.py to see where to add support for this. Any suggestions are appreciated. Thanks!

[Technologicat]

Hi @Daniel-Mock ! Sorry for the late reply, I've been working on another project.

Fixtures break ZoP's explicit is better than implicit guideline: they are imported implicitly by pytest.

So, if we want to support them, what we need is some logic to statically detect which fixtures are in scope (to know which names refer to fixtures), and where they are defined (so we can add the uses-edges). This is probably a rather large and complex change, and it's beyond the scope of what I need pyan to do. But if you want to have a go at adding it:

Detecting the use of a fixture

• As in your example, the use of a fixture may be declared by a decorator.
  • In the Python AST, decorators are attached to the function definition node, so they are analyzed in analyze_functiondef. You'll need to specifically consider how typical decorator invocations looks like when using pytest's fixtures, and get the fixture names from those decorator invocations. There are likely several formats. Maybe only support the most common ones?
  • For example, look at the AST for the code @pytest.mark.usefixtures("fixture_1") to see how to extract the name fixture_1. Off the top of my head, that's an ast.Call where func consists of two nested ast.Attribute nodes, and the string argument is an ast.Constant. An AST dumper is useful here. (The one linked here is based on Alex Leone's original, but has been customized to produce ANSI-colored output, with mostly PEP8-compliant indentation.)
  • You'll first want to detect that the attribute comes from pytest, to know whether it's worth looking at more closely. Here's a snippet that converts a nested ast.Attribute AST into a dotted name as a string, which can then be processed easily:

    def attr_ast_to_dotted_name(tree):
        # Input is like:
        #     (a.b).thing
        #     ((a.b).c).thing
        #     ...
        if type(tree) is not ast.Attribute:
            raise TypeError
        acc = []
        def recurse(tree):
            acc.append(tree.attr)
            if type(tree.value) is ast.Attribute:
                recurse(tree.value)
            elif type(tree.value) is ast.Name:
                acc.append(tree.value.id)
            else:
                raise NotImplementedError
        recurse(tree)
        return ".".join(reversed(acc))

(That snippet comes from an old development version of mcpyrate.) If you want to handle the fully general case (not only ast.Attribute and ast.Name), you'll need a Python AST unparser, such as this one. (You can strip away the AST marker stuff, that's a mcpyrate thing.) There are also suggestions for unparser libraries in the AST docs. From a cursory look, parso looks promising and should be up to date, but I haven't used it myself. Alternatively, you could also skip the unparsing, and analyze the AST directly - whichever is easier.

• Fixtures can also arrive by dependency injection, as in your test_1 above. So function parameter names must be matched against all fixtures that are in scope at that point.

• You'll need to follow pytest's fixture scoping rules to determine which fixture a name refers to, if there are several fixtures with the same name.

Gathering the definitions of fixtures

• A function definition decorated by @pytest.fixture is a fixture. So in analyze_functiondef, look at the decorators.

• pytest implicitly imports fixtures from conftest.py files, when those files are placed in an appropriate location. See the pytest docs on fixtures. So before the analysis begins, look at the directory structure, and then add those files into the set of files being analyzed. Maybe in the constructor, or maybe outside the analyzer.