JP/JSR 3.22.1: excessive time to analyze methods with a lot of lines

[corradomio]

The problem with methods containing a lot of code lines is not resolved yet.

For example, in "commons-lang3", file "ArrayUtilsTest", the method "void testSameLengthAll()" is composed by ~1330 lines. The analysis requires A LOT of time!

[maartenc]

Coud you build the HEAD revision of the master branch and try again? It contains some performance improvements that are not released yet.

[corradomio]

Ok, I will try. Thanks

On Thu, Jun 10, 2021 at 3:10 PM Maarten Coene @.***> wrote:

Coud you build the HEAD revision of the master branch and try again? It contains some performance improvements that are not released yet.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/javaparser/javaparser/issues/3292#issuecomment-858530793, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUJJIE2ISVWBZ3X4NPSFN3TSCMQRANCNFSM46OICA5A .

[maartenc]

Did it solve your issue @corradomio ?

[corradomio]

I am sorry, but I have not checked yet :-( Currently, my application ""aborts"" automatically the analysis IF it spent excessive time on a single method or on a file. For this reason, for now, this problem has a low priority.

On Tue, Jun 15, 2021 at 1:16 PM Maarten Coene @.***> wrote:

Did it solve your issue @corradomio https://github.com/corradomio ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/javaparser/javaparser/issues/3292#issuecomment-861333817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFUJJIHD6GQMMVWM2JEELI3TS4K63ANCNFSM46OICA5A .

[michaelhkay]

Just to report that I have the same issue. I haven't explored it fully because it's not a complete show-stopper, and because it's difficult to isolate from a complex build. I do know that it's a regression since 1.16. I have a class with a static initialisation block that's about 600 lines of this kind of thing:

static {
        data[0] = (byte) 0;
        Arrays.fill(data, 1, 9, (byte) 8);
        Arrays.fill(data, 9, 11, (byte) 9);
        Arrays.fill(data, 11, 13, (byte) 8);
        data[13] = (byte) 9;
        Arrays.fill(data, 14, 32, (byte) 8);
        Arrays.fill(data, 32, 45, (byte) 9);
        Arrays.fill(data, 45, 47, (byte) 27);
        data[47] = (byte) 9;
        ...
}

and it takes about a minute to process.

In case anyone wants to investigate it, there's a copy of the module here:

https://saxonica.plan.io/projects/saxon/repository/he/revisions/master/entry/latest10/hej/net/sf/saxon/serialize/charcode/XMLCharacterData.java

[maartenc]

Did you try with HEAD revision of the master branch @michaelhkay ?

[michaelhkay]

@maartenc No, sorry, I'm afraid I've had a lot of trouble building a working project from a direct checkout. Got into a tangle of dependencies.

[maartenc]

Could you also post a snippet of your javaparser code to illustrate the problem @michaelhkay ?

[michaelhkay]

I'm invoking the JavaParser like this:

        CompilationUnit cu = StaticJavaParser.parse(filename);
        XmlPrinterMHK printer = new XmlPrinterMHK(symbolSolver, true);
        File xmlOut = new File(xmlDir, filename.toString().replace(".java", ".xml"));
        System.err.println("Writing to " + xmlOut);
        PrintStream p = new PrintStream(xmlOut);
        p.println(printer.output(cu));

where XmlPrinterMHK is my own variant of XmlPrinter as follows:

public class XmlPrinterMHK {

    private JavaSymbolSolver solver;
    private final boolean outputNodeType;

    public XmlPrinterMHK(JavaSymbolSolver solver, boolean outputNodeType) {
        this.outputNodeType = outputNodeType;
        this.solver = solver;
    }

    public String output(Node node) {
        StringBuilder sb = new StringBuilder();
        output(node, "root", 0, sb);
        return sb.toString();
    }

    public void output(Node node, String name, int level, StringBuilder builder) {
        assertNotNull(node);
        NodeMetaModel metaModel = node.getMetaModel();
        List<PropertyMetaModel> allPropertyMetaModels = metaModel.getAllPropertyMetaModels();
        List<PropertyMetaModel> attributes = allPropertyMetaModels.stream().filter(PropertyMetaModel::isAttribute).filter(PropertyMetaModel::isSingular).collect(toList());
        List<PropertyMetaModel> subNodes = allPropertyMetaModels.stream().filter(PropertyMetaModel::isNode).filter(PropertyMetaModel::isSingular).collect(toList());
        List<PropertyMetaModel> subLists = allPropertyMetaModels.stream().filter(PropertyMetaModel::isNodeList).collect(toList());

        builder.append("<").append(name);
        if (outputNodeType) {
            builder.append(attribute("nodeType", metaModel.getTypeName()));
        }

        for (PropertyMetaModel attributeMetaModel : attributes) {
            builder.append(attribute(attributeMetaModel.getName(), attributeMetaModel.getValue(node).toString()));
        }

        if (node instanceof MethodCallExpr) {
            try {
                String returnType = ((MethodCallExpr) node).resolve().getReturnType().describe();
                builder.append(attribute("RETURN", returnType));
            } catch (Exception e) {
                // no action
            }
        }
        if (node instanceof NameExpr) {
            try {
                ResolvedValueDeclaration decl = ((NameExpr) node).resolve();
                // Detect a reference to a value in an enumeration class
                if (decl.isField() && decl.asField().declaringType().isEnum()) {
                    builder.append(attribute("ENUM_TYPE", decl.asField().declaringType().getQualifiedName()));
                }
                ResolvedType resolvedType = ((NameExpr) node).calculateResolvedType();
                builder.append(attribute("RESOLVED_TYPE", resolvedType.describe()));
            } catch (Exception e) {
                builder.append(attribute("UNRESOLVED", node.toString()));
            }
        } else if (node instanceof FieldAccessExpr) {
            // try to determine if we are accessing a static field of a generic type (notably Feature<T>)
            try {
                String returnType = ((FieldAccessExpr) node).resolve().getType().describe();
                builder.append(attribute("RETURN", returnType));
            } catch (Exception e) {
                // no action
            }
            try {

                ResolvedType resolvedType = ((NodeWithScope<?>) node).getScope().calculateResolvedType();
                builder.append(attribute("RESOLVED_TYPE", resolvedType.describe()));
                if (resolvedType instanceof ReferenceTypeImpl) {
                    Optional<ResolvedReferenceTypeDeclaration> decl = ((ReferenceTypeImpl) resolvedType).getTypeDeclaration();
                    if (decl.isPresent() && !decl.get().getTypeParameters().isEmpty()) {
                        ResolvedFieldDeclaration fieldDecl = decl.get().getField(((FieldAccessExpr) node).getNameAsString());
                        if (fieldDecl.isStatic()) {
                            ResolvedType fieldType = fieldDecl.asField().getType();
                            fieldType.describe();
                            builder.append(attribute("FIELD_TYPE", fieldType.describe()));
                        }
                    }
                }
            } catch (Exception e) {
                builder.append(attribute("UNRESOLVED", ((NodeWithScope<?>) node).getScope().toString()));
            }
        } else if (node instanceof NodeWithScope) {
            try {
                ResolvedType resolvedType = ((NodeWithScope<?>) node).getScope().calculateResolvedType();
                //System.err.println("Resolved symbol: " + ((NodeWithScope<?>) node).getScope().toString() + " => " + resolvedType.describe());
                builder.append(attribute("RESOLVED_TYPE", resolvedType.describe()));
            } catch (RuntimeException e) {
                builder.append(attribute("UNRESOLVED", ((NodeWithScope<?>) node).getScope().toString()));
            }
        } else if (node instanceof NodeWithOptionalScope) {
            try {
                Optional<com.github.javaparser.ast.expr.Expression> optScope = ((NodeWithOptionalScope) node).getScope();
                if (optScope.isPresent()) {
                    ResolvedType resolvedType = optScope.get().calculateResolvedType();
                    //System.err.println("Resolved symbol: " + ((NodeWithScope<?>) node).getScope().toString() + " => " + resolvedType.describe());
                    builder.append(attribute("RESOLVED_TYPE", resolvedType.describe()));
                } else if (node instanceof MethodCallExpr) {
                    String qualifiedMethodName = ((MethodCallExpr) node).resolve().getQualifiedName();
                    int lastDot = qualifiedMethodName.lastIndexOf(".");
                    builder.append(attribute("DECLARING_TYPE", qualifiedMethodName.substring(0, lastDot)));
                }
            } catch (RuntimeException e) {
                builder.append(attribute("UNRESOLVED", ((NodeWithOptionalScope<?>) node).getScope().toString()));
            }
        } else if (node instanceof com.github.javaparser.ast.type.Type) {
            try {
                ResolvedType resolvedType = ((com.github.javaparser.ast.type.Type) node).resolve();
                builder.append(attribute("RESOLVED_TYPE", resolvedType.describe()));
            } catch (RuntimeException e) {
                //System.err.println("Unresolved: " + node.toString() + " at " + node.getRange());
                builder.append(attribute("UNRESOLVED", node.toString()));
            }
        }

        builder.append(">");

        for (PropertyMetaModel subNodeMetaModel : subNodes) {
            Node value = (Node) subNodeMetaModel.getValue(node);
            if (value != null) {
                output(value, subNodeMetaModel.getName(), level + 1, builder);
            }
        }

        for (PropertyMetaModel subListMetaModel : subLists) {
            NodeList<? extends Node> subList = (NodeList<? extends Node>) subListMetaModel.getValue(node);
            if (subList != null) {
                String listName = subListMetaModel.getName();
                if (subList.isEmpty()) {
                    if (listName.equals("typeArguments")) {
                        // MHK: this is the only case we know of where an empty list differs from an absent list, e.g. new HashSet<>().
                        builder.append("<").append(listName).append("/>");
                    }
                } else {
                    builder.append("<").append(listName).append(">");
                    String singular = listName.substring(0, listName.length() - 1);
                    for (Node subListNode : subList) {
                        output(subListNode, singular, level + 1, builder);
                    }
                    builder.append(close(listName));
                }
            }
        }
        builder.append(close(name));
    }

    private static String close(String name) {
        return "</" + name + ">";
    }

    private static String escape(String value) {
        return value.replace("&", "&amp;").replace("<", "&lt;").replace("'", "&apos;");
    }

    private static String attribute(String name, String value) {
        return " " + name + "='" + escape(value) + "'";
    }

}

The code includes the results of a lot of trial and error and could no doubt be greatly improved if I knew what I was doing - but the APIs aren't exactly well documented. I'm basically trying to get extra type information from the symbol solver, and add it to the tree while serializing it as XML.