Add GitHub action to run SonarQube for METveiwer pull requests and feature branches

JohnHalleyGotway commented 7 months ago

Describe the New Feature

This issue is to add a new SonarQube workflow to GitHub actions to automate the static code analysis for all pull requests. In addition, add a manual trigger workflow dispatch option where the reference branch can be manually specified.

Recommend adding this workflow to both the develop branch and the current main_v*so that the workflow dispatch option can be made available.

Recommend pushing results to a new SonarQube project named METviewer GHA at needham.rap.ucar.edu.

Recommend having the workflow report bad status if the number of SonarQube findings are increased relative to the SonarQube reference.

See issue dtcenter/MET#2379 and corresponding PR's as an example. Scanning the METviewer software may be more complicated that scanning a python-only repo. We may need to do this inside a Docker container, as we've done for the MET C++ code.

See instructions from @TatianaBurek here: https://github.com/dtcenter/METplus-Internal/issues/35#issuecomment-2047821434

Acceptance Testing

List input data types and sources. Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

[ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Needed for the Air Force - 2771024

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[x] Select component(s)
[x] Select priority
[x] Select requestor(s)

Projects and Milestone

[x] Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
[x] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

[x] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

The following SonarQube issues are closely related:

[x] dtcenter/METplus#2537
[x] dtcenter/MET#2379
[x] dtcenter/METviewer#521
[x] dtcenter/METplotpy#429
[x] dtcenter/METcalcpy#366
[x] dtcenter/METdataio#289

New Feature Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

JohnHalleyGotway commented 7 months ago

Copied from dtcenter/METplus-Internal#35.

How to run SonarQube report for METviewer System requirements:

Java 11
Ant
sonarqube-ant-task-2.7.1.1951.jar build_sonar.xml.txt

Steps:

Clone METviewer from GItHub repository
Place build_sonar.xml under the METviewer directory
Define the SonarQube global/project properties and the Scanner - lines 13 - 39 in build_sonar.xml: sonar.host.url sonar.projectKey sonar.projectName sonar.login sonar.password sonar.java.jdkHome path to sonarqube-ant-task-2.7.1.1951.jar
set JAVA_HOME: JAVA_HOME= export JAVA_HOME
run Ant ant sonar -buildfile build_sonar.xml -Ddb.management.system=mariadb

JohnHalleyGotway commented 7 months ago

Ran the following steps to test.

/Volumes/d1/projects/METviewer/METviewer-develop/docker
docker build -t metviewer_local .
docker run -it --rm metviewer_local /bin/bash
export SONAR_SCANNER_VERSION=5.0.1.3006
mkdir -p $HOME/.sonar
curl -sSLo $HOME/.sonar/sonar-scanner.zip https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-${SONAR_SCANNER_VERSION}-linux.zip
unzip -o $HOME/.sonar/sonar-scanner.zip -d $HOME/.sonar/
echo export PATH="$HOME/.sonar/sonar-scanner-${SONAR_SCANNER_VERSION}-linux/bin:\$PATH" >> $HOME/.bashrc
source ~/.bashrc
cd /METviewer
# Create sonar-project.properties including:
#   sonar.java.libraries=dist/lib
#   sonar.java.binaries=dist/metviewer/WEB-INF/classes
sonar-scanner

And that produces this result: https://needham.rap.ucar.edu/dashboard?id=METviewer

JohnHalleyGotway commented 7 months ago

After merging these changes into the develop and main_v5.1 branches, I re-configured the DockerHub automated builds:

For metviewer, I updated the build settings to define the Build Context as /internal/scripts/docker.
For metviewer-singularity, I updated the build settings to define the Build Context as /internal/scripts/docker and the Dockerfile location as Dockerfile.apptainer (previously it was Dockerfile_for_singularity.
I manually triggered a rebuild of the develop branch for both and note the dtcenter/metviewer build and the dtcenter/metviewer-singularity build were both successful.
I also manually triggered rebuilds of the main_v5.1 branch.

Manually reran SonarQube with the following commands:

docker pull dtcenter/metviewer:develop
docker run -it --rm dtcenter/metviewer:develop /bin/bash
cd /METviewer
git branch # confirm that its develop
# Create build_sonar.xml, as described in above comments
wget https://repo1.maven.org/maven2/org/sonarsource/scanner/ant/sonarqube-ant-task/2.7.1.1951/sonarqube-ant-task-2.7.1.1951.jar
mv sonarqube-ant-task-2.7.1.1951.jar lib/.
ant sonar -buildfile build_sonar.xml -Ddb.management.system=mariadb

Used these settings in build_sonar.xml:

    <property name="sonar.host.url" value="https://needham.rap.ucar.edu" />
    <property name="sonar.projectKey" value="METviewer" />
    <property name="sonar.projectName" value="METviewer" />
    <property name="sonar.projectVersion" value="6.0.0-BETA5-DEV" />
    <property name="sonar.branch.name" value="feature_521_sonarqube_ant_build" />
    <property name="sonar.sources" value="java" />
    <property name="sonar.sourceEncoding" value="UTF-8" />
...
        <taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
            <!-- Update the following line, or put the "sonar-ant-task-*.jar" file in your "$HOME/.ant/lib" folder -->
            <classpath path="/METviewer/lib/sonarqube-ant-task-2.7.1.1951.jar" />
        </taskdef>

This process fails with a Java version issue.

[root@82fd7973953d METviewer]# ant sonar -buildfile build_sonar.xml -Ddb.management.system=mariadb
Buildfile: /METviewer/build_sonar.xml

init:
   [delete] Deleting directory /METviewer/build
    [mkdir] Created dir: /METviewer/build
   [delete] Deleting directory /METviewer/dist
    [mkdir] Created dir: /METviewer/dist

test.db.management.system:

compile:
    [javac] Compiling 88 source files to /METviewer/build
    [javac] warning: Implicitly compiled files were not subject to annotation processing.
    [javac]   Use -proc:none to disable annotation processing or -implicit to specify a policy for implicit compilation.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 warning

sonar:
[sonar:sonar] Apache Ant(TM) version 1.9.4 compiled on November 5 2018
[sonar:sonar] SonarQube Ant Task version: 2.7.1.1951
[sonar:sonar] Loaded from: file:/METviewer/lib/sonarqube-ant-task-2.7.1.1951.jar
[sonar:sonar] User cache: /root/.sonar/cache
[sonar:sonar] Default locale: "en_US", source code encoding: "UTF-8"

BUILD FAILED
/METviewer/build_sonar.xml:39: java.lang.UnsupportedClassVersionError: org/sonar/batch/bootstrapper/EnvironmentInformation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at org.sonarsource.scanner.api.internal.IsolatedClassloader.loadClass(IsolatedClassloader.java:82)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.sonarsource.scanner.api.internal.batch.DefaultBatchFactory.createBatch(DefaultBatchFactory.java:32)
    at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
    at com.sun.proxy.$Proxy2.execute(Unknown Source)
    at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
    at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
    at org.sonarsource.scanner.ant.SonarQubeTask.launchAnalysis(SonarQubeTask.java:113)
    at org.sonarsource.scanner.ant.SonarQubeTask.execute(SonarQubeTask.java:94)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:435)
    at org.apache.tools.ant.Target.performTasks(Target.java:456)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
    at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
    at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1248)
    at org.apache.tools.ant.Main.runBuild(Main.java:851)
    at org.apache.tools.ant.Main.startAnt(Main.java:235)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)

Total time: 3 seconds

dtcenter / METviewer