acanda / eclipse-pmd

eclipse-pmd has been moved to
https://github.com/eclipse-pmd/eclipse-pmd
Other
21 stars 10 forks source link

Build hangs on project with SQL files #16

Closed ianbrandt closed 10 years ago

ianbrandt commented 10 years ago

I'm trying to enable the PMD nature on a Maven module project that contains 540 .sql files and 290 .java files. When I do this Building workspace hangs at 43% on "Setting classpath containers: Invoking 'PMD Builder' on '/db'." The Eclipse process has the CPU pegged at 700+% (4 cores, hyperthreaded, so 800% is the max). After running for over an hour the build eventually fails having run out of "Java heap space" (-Xmx1536m -XX:MaxPermSize=512m). After the failure I can remove the PMD nature from the project in question and restart Eclipse.

I took occasional stack dumps while it was running, and a single ever-present stack for PLSQLParser stood out:

"Worker-111" #596 prio=5 os_prio=31 tid=0x000000010096d800 nid=0xfbab runnable [0x000000012db31000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Throwable.fillInStackTrace(Native Method)
    at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
    - locked <0x00000007a296da60> (a java.io.IOException)
    at java.lang.Throwable.<init>(Throwable.java:265)
    at java.lang.Exception.<init>(Exception.java:66)
    at java.io.IOException.<init>(IOException.java:58)
    at java.io.BufferedReader.ensureOpen(BufferedReader.java:122)
    at java.io.BufferedReader.read(BufferedReader.java:278)
    - locked <0x000000077afee790> (a java.io.InputStreamReader)
    at net.sourceforge.pmd.lang.ast.SimpleCharStream.FillBuff(SimpleCharStream.java:114)
    at net.sourceforge.pmd.lang.ast.SimpleCharStream.readChar(SimpleCharStream.java:196)
    at net.sourceforge.pmd.lang.ast.SimpleCharStream.BeginToken(SimpleCharStream.java:136)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParserTokenManager.getNextToken(PLSQLParserTokenManager.java:4347)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.getNextToken(PLSQLParser.java:39826)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.Skip2NextTerminator(PLSQLParser.java:9001)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.SqlStatement(PLSQLParser.java:9250)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.UnlabelledStatement(PLSQLParser.java:9355)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.Statement(PLSQLParser.java:9974)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.View(PLSQLParser.java:22036)
    at net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.Input(PLSQLParser.java:182)
    at net.sourceforge.pmd.lang.plsql.PLSQLParser.parse(PLSQLParser.java:52)
    at net.sourceforge.pmd.SourceCodeProcessor.parse(SourceCodeProcessor.java:95)
    at net.sourceforge.pmd.SourceCodeProcessor.processSource(SourceCodeProcessor.java:142)
    at net.sourceforge.pmd.SourceCodeProcessor.processSourceCode(SourceCodeProcessor.java:80)
    at ch.acanda.eclipse.pmd.builder.Analyzer.analyze(Analyzer.java:72)
    at ch.acanda.eclipse.pmd.builder.PMDBuilder.analyze(PMDBuilder.java:75)
    at ch.acanda.eclipse.pmd.builder.PMDBuilder$ResourceVisitor.visit(PMDBuilder.java:99)
    at org.eclipse.core.internal.resources.Resource$2.visit(Resource.java:126)
    at org.eclipse.core.internal.resources.Resource$1.visitElement(Resource.java:85)
    at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:82)
    at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
    at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
    at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
    at org.eclipse.core.internal.watson.ElementTreeIterator.doIteration(ElementTreeIterator.java:86)
    at org.eclipse.core.internal.watson.ElementTreeIterator.iterate(ElementTreeIterator.java:127)
    at org.eclipse.core.internal.resources.Resource.accept(Resource.java:95)
    at org.eclipse.core.internal.resources.Resource.accept(Resource.java:52)
    at org.eclipse.core.internal.resources.Resource.accept(Resource.java:124)
    at org.eclipse.core.internal.resources.Resource.accept(Resource.java:108)
    at ch.acanda.eclipse.pmd.builder.PMDBuilder.fullBuild(PMDBuilder.java:62)
    at ch.acanda.eclipse.pmd.builder.PMDBuilder.build(PMDBuilder.java:48)
    at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:733)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:206)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:246)
    at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:299)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:302)
    at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:358)
    at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:381)
    at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:143)
    at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:241)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53)

Assuming I've specified the patterns correctly my rule set excludes all but .java files, and I don't reference any of the "rulesets/plsql/*" rule sets:

<?xml version="1.0" encoding="UTF-8" ?>
<ruleset name="etpruleset" xmlns="http://pmd.sourceforge.net/ruleset/2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://pmd.sourceforge.net/ruleset/2.0.0 http://pmd.sourceforge.net/ruleset_2_0_0.xsd">

    <description>My Ruleset</description>

    <exclude-pattern>**/*</exclude-pattern>
    <include-pattern>**/*.java</include-pattern>

    <rule ref="rulesets/java/basic.xml" />
[...]

When I run mvn pmd:pmd on this project it completes without issue in a few seconds.

acanda commented 10 years ago

PMD uses regular expressions to define the include and exclude pattern and (unfortunately) silently ignores an invalid pattern. In your case both the exclude and the includes pattern are invalid and that is why PMD analyses the SQL files in the first place. You might want to change your patterns to the following:

    <exclude-pattern>.*</exclude-pattern>
    <include-pattern>.*\.java$</include-pattern>

I tried to reproduce the problem with my own SQL files but was unable to do so. My currently best guess for the reason why mvn pmd:pmd runs without issue and eclipse-pmd hangs is that mvn pmd:pmd does not actually analyse the SQL files. Please make sure that

Your maven configuration should look like this:

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-pmd-plugin</artifactId>
    <version>3.1</version>
    <configuration>
        <targetJdk>1.7</targetJdk>
        <rulesets>
            <ruleset>${basedir}/pmd.xml</ruleset>
        </rulesets>
        <verbose>true</verbose>
        <includes>
          <include>**/*</include>
        </includes>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>net.sourceforge.pmd</groupId>
            <artifactId>pmd</artifactId>
            <version>5.1.1</version>
        </dependency>
    </dependencies>
  </plugin>

Please let me know if the above maven configuration also hangs when analysing your SQL files. This would help me find out if this is a problem with eclipse-pmd or PMD itself.

ianbrandt commented 10 years ago

Thanks for looking into this. Indeed I was mistakenly using Ant-like patterns in my rule set.

I've added the include to my maven-pmd-plugin configuration, leaving the rest as it was. Per mvn help:effective-pom:

<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
        </resource>
        <resource>
            <directory>src/main/sql</directory>
        </resource>
    </resources>
    <testResources>
        <testResource>
            <directory>src/it/resources</directory>
        </testResource>
    </testResources>
    <plugins>
        <plugin>
            <artifactId>maven-pmd-plugin</artifactId>
            <version>3.1</version>
            <dependencies>
                <dependency>
                    <groupId>net.sourceforge.pmd</groupId>
                    <artifactId>pmd</artifactId>
                    <version>5.1.1</version>
                    <scope>compile</scope>
            </dependency>
        </dependencies>
        <configuration>
            <targetJdk>1.6</targetJdk>
            <rulesets>
                <ruleset>file:///Users/ibrandt/myproject/db/../pmd.xml</ruleset>
            </rulesets>
            <includes>
                <include>**/*</include>
            </includes>
            <includeTests>true</includeTests>
            <verbose>true</verbose>
        </configuration>
        </plugin>
        [...]

I removed the exclude and include from my rule set:

<?xml version="1.0" encoding="UTF-8" ?>
<ruleset name="myruleset" xmlns="http://pmd.sourceforge.net/ruleset/2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://pmd.sourceforge.net/ruleset/2.0.0 http://pmd.sourceforge.net/ruleset_2_0_0.xsd">

    <description>My Ruleset</description>

    <rule ref="rulesets/java/basic.xml" />
    <rule ref="rulesets/java/braces.xml" />
    [...]

It doesn't seem to get hung up on the SQL:

$ mvn pmd:pmd
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building DB 7.2.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-pmd-plugin:3.1:pmd (default-cli) @ db ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.238s
[INFO] Finished at: Sun Jul 13 08:53:55 PDT 2014
[INFO] Final Memory: 18M/81M
[INFO] ------------------------------------------------------------------------

Of note the SQL files are in an added resource folder, "src/main/sql". This is also a source folder in Eclipse. Might the maven-pmd-plugin not scan added resource folders, where eclipse-pmd does on account of them being source folders?

ianbrandt commented 10 years ago

I added the correct exclude patterns to my rule set:

    <exclude-pattern>.*</exclude-pattern>
    <include-pattern>.*\.java$</include-pattern>

I can now enable the PMD nature on my DB project, and the build completes just fine.

acanda commented 10 years ago

I don't know if the maven-pmd-plugin scans resource folders. You could run PMD from the command line and specify the resource folder explicitly with -d <path> to make sure the SQL files are included.

ianbrandt commented 10 years ago

I'm going to go ahead and close this as eclipse-pmd is now working fine for me with .sql files properly excluded. I assume the root performance issue is with the PLSQLParser upstream. I'll file an issue for that with PMD directly.

sturton commented 10 years ago

The PLSQL Parser was inherited / migrated from PLDoc where it concentrates on PLSQL, but tries to support other commands that might be found in a SQL script file:- SQL*Plus, DDL, GRANTS, etc.

It does attempt to support Table and View DDL, but that support has not been maintained or updated, simply because other tools do Table and View documentation much, much better - I use maven-schemaspy-plugin .

I infer from a look at the stack trace that the parser is having problems parsing a View DDL statement " ... net.sourceforge.pmd.lang.plsql.ast.PLSQLParser.View ... "

I will have a look at the parser - it would help if you could post the problem View definition.