search

Sage-Bionetworks / sage-monorepo

Where OpenChallenges, Schematic, and other Sage open source apps are built
https://sage-bionetworks.github.io/sage-monorepo/
Apache License 2.0
23 stars 12 forks source link

[Story] Enable Sonar for the project `openchallenges-edam-etl` #2565

Closed tschaffter closed 6 months ago

tschaffter commented 6 months ago

What product(s) is this story for?

OpenChallenges

As a user, I want

As a Developer, I want the CI/CD workflow to scan the changes made to the project openchallenges-edam-etl when I work in a PR so that I can get useful insights about bugs, code smells and known vulnerabilities.

Description

  1. Create the project openchallenges-edam-etl on SonarCloud.io
  2. Add the task sonar to the project openchallenges-edam-etl

Acceptance criteria

No response

Tasks

No response

Anything else?

No response

Have you linked this story to a GitHub Project?

tschaffter commented 6 months ago

How to create a project on SonarCloud.io

  1. Login on https://sonarcloud.io
  2. Click on the "+" button in the navbar, then select "Analyze new project"
  3. Click on the link "Setup a monorepo"
  4. Organization: select "Sage Bionetworks"
  5. Repository: select "sage-monorepo"
  6. Click on the button "Add new project"
  7. Project Key: specify the name of the project as defined in the file project.json of the corresponding project. For example, "openchallenges-edam-etl".
  8. Display Name: keep the auto-generated value, which should be the same as the Project Key.
  9. Click on the button "Save configuration"
  10. The new code for this project will be based on: select "Previous version"
  11. Click on the button "Create project"
tschaffter commented 6 months ago

How to add the task sonar to the project

I copied the task sonar from apps/schematic/api/project.json, another Python project, and pasted it to apps/openchallenges-edam-etl/project.json.

Testing the task sonar for the project openchallenges-edam-etl locally:

vscode@dee30b82cf44:/workspaces/sage-monorepo$ export SONAR_TOKEN=YOUR_TOKEN
vscode@dee30b82cf44:/workspaces/sage-monorepo$ nx sonar openchallenges-edam-etl

> nx run openchallenges-edam-etl:sonar

INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 5.0.1.3006
INFO: Java 17.0.9 Private Build (64-bit)
INFO: Linux 4.14.336-255.557.amzn2.x86_64 amd64
INFO: User cache: /opt/sonar-scanner/.sonar/cache
INFO: Analyzing on SonarCloud
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=427ms
INFO: Server id: 1BD809FA-AWHW8ct9-T_TB3XqouNu
INFO: User cache: /opt/sonar-scanner/.sonar/cache
INFO: Loading required plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=412ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=1010ms
INFO: Load project settings for component key: 'openchallenges-edam-etl'
INFO: Load project settings for component key: 'openchallenges-edam-etl' (done) | time=376ms
INFO: Process project properties
INFO: Project key: openchallenges-edam-etl
INFO: Base dir: /workspaces/sage-monorepo/apps/openchallenges/edam-etl
INFO: Working dir: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork
INFO: Load project branches
INFO: Load project branches (done) | time=386ms
INFO: Check ALM binding of project 'openchallenges-edam-etl'
INFO: Detected project binding: BOUND
INFO: Check ALM binding of project 'openchallenges-edam-etl' (done) | time=357ms
INFO: Load project pull requests
INFO: Load project pull requests (done) | time=367ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=4ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=458ms
INFO: Load active rules
INFO: Load active rules (done) | time=6417ms
INFO: Organization key: sage-bionetworks
INFO: Preprocessing files...
INFO: 3 languages detected in 10 preprocessed files
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Loading plugins for detected languages
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=235ms
INFO: Load project repositories
INFO: Load project repositories (done) | time=395ms
INFO: Indexing files...
INFO: Project configuration:
INFO:   Excluded sources: **/build-wrapper-dump.json
INFO: 10 files indexed
INFO: Quality profile for docker: Sonar way
INFO: Quality profile for json: Sonar way
INFO: Quality profile for py: Sage way
INFO: ------------- Run sensors on module openchallenges-edam-etl
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=372ms
INFO: Sensor cache enabled
INFO: Load sensor cache
INFO: Load sensor cache (404) | time=276ms
INFO: Sensor Python Sensor [python]
WARN: Your code is analyzed as compatible with all Python 3 versions by default. You can get a more precise analysis by setting the exact Python version in your configuration via the parameter "sonar.python.version"
INFO: Starting global symbols computation
INFO: 2 source files to be analyzed
INFO: 2/2 source files have been analyzed
INFO: Starting rules execution
INFO: 2 source files to be analyzed
INFO: 2/2 source files have been analyzed
INFO: The Python analyzer was able to leverage cached data from previous analyses for 0 out of 2 files. These files were not parsed.
INFO: Sensor Python Sensor [python] (done) | time=1600ms
INFO: Sensor Cobertura Sensor for Python coverage [python]
WARN: No report was found for sonar.python.coverage.reportPaths using pattern ./coverage.xml
INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=78ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=23ms
INFO: Sensor Python HTML templates processing [securitypythonfrontend]
INFO: Sensor Python HTML templates processing [securitypythonfrontend] (done) | time=11ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=3ms
INFO: Sensor IaC CloudFormation Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC CloudFormation Sensor [iac] (done) | time=35ms
INFO: Sensor IaC AzureResourceManager Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC AzureResourceManager Sensor [iac] (done) | time=147ms
INFO: Sensor IaC Docker Sensor [iac]
INFO: 1 source file to be analyzed
INFO: 1/1 source file has been analyzed
INFO: Sensor IaC Docker Sensor [iac] (done) | time=135ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=7ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=1ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=1ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ir/java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=2ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ir/python
INFO: Analyzing 1 functions to detect bugs.
INFO: Sensor pythonbugs [dbd] (done) | time=220ms
INFO: Sensor TextAndSecretsSensor [text]
INFO: Available processors: 8
INFO: Using 8 threads for analysis.
INFO: 4 source files to be analyzed
INFO: 4/4 source files have been analyzed
INFO: Sensor TextAndSecretsSensor [text] (done) | time=544ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Enabled taint analysis rules: S2076, S2078, S2083, S2091, S2631, S3649, S5131, S5135, S5144, S5145, S5146, S5147, S5334, S5883, S6096, S6173, S6287, S6350, S6384, S6390, S6398, S6399, S6547, S6549
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ucfg2/java
INFO: Read 0 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.002
INFO: Load UCFGs: Starting
INFO: Load UCFGs: Time spent was 00:00:00.000
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.003
INFO: No UCFGs have been included for analysis.
INFO: java security sensor: Time spent was 00:00:00.007
INFO: Sensor JavaSecuritySensor [security] (done) | time=21ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Enabled taint analysis rules: S2076, S2078, S2083, S2091, S2631, S3649, S5131, S5135, S5144, S5145, S5146, S5334, S5883, S6096, S6173, S6287, S6350, S6399, S6639, S6641
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/ucfg2/cs
INFO: Read 0 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.000
INFO: Load UCFGs: Starting
INFO: Load UCFGs: Time spent was 00:00:00.000
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.000
INFO: No UCFGs have been included for analysis.
INFO: csharp security sensor: Time spent was 00:00:00.001
INFO: Sensor CSharpSecuritySensor [security] (done) | time=2ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Enabled taint analysis rules: S2076, S2078, S2083, S2091, S2631, S3649, S5131, S5135, S5144, S5145, S5146, S5334, S5335, S5883, S6173, S6287, S6350
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ucfg2/php
INFO: Read 0 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.000
INFO: Load UCFGs: Starting
INFO: Load UCFGs: Time spent was 00:00:00.000
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.000
INFO: No UCFGs have been included for analysis.
INFO: php security sensor: Time spent was 00:00:00.000
INFO: Sensor PhpSecuritySensor [security] (done) | time=1ms
INFO: Sensor PythonSecuritySensor [security]
INFO: Enabled taint analysis rules: S2076, S2078, S2083, S2091, S2631, S3649, S5131, S5135, S5144, S5145, S5146, S5147, S5334, S5496, S6287, S6350, S6639, S6680, S6776, S6839
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ucfg2/python
INFO: Read 209 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.079
INFO: Load UCFGs: Starting
INFO: Reading UCFGs from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ucfg2/python
INFO: Load UCFGs: Time spent was 00:00:00.264
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.344
INFO: Analyzing 1058 UCFGs to detect vulnerabilities.
INFO: Check cache: Starting
INFO: Load cache: Starting
INFO: Load cache: Time spent was 00:00:00.000
INFO: Check cache: Time spent was 00:00:00.000
INFO: Create runtime call graph: Starting
INFO: Variable Type Analysis #1: Starting
INFO: Create runtime type propagation graph: Starting
INFO: Create runtime type propagation graph: Time spent was 00:00:00.020
INFO: Run SCC (Tarjan) on 2168 nodes: Starting
INFO: Run SCC (Tarjan) on 2168 nodes: Time spent was 00:00:00.020
INFO: Tarjan found 2168 strongly connected components
INFO: Propagate runtime types to strongly connected components: Starting
INFO: Propagate runtime types to strongly connected components: Time spent was 00:00:00.016
INFO: Variable Type Analysis #1: Time spent was 00:00:00.060
INFO: Variable Type Analysis #2: Starting
INFO: Create runtime type propagation graph: Starting
INFO: Create runtime type propagation graph: Time spent was 00:00:00.008
INFO: Run SCC (Tarjan) on 2166 nodes: Starting
INFO: Run SCC (Tarjan) on 2166 nodes: Time spent was 00:00:00.004
INFO: Tarjan found 2166 strongly connected components
INFO: Propagate runtime types to strongly connected components: Starting
INFO: Propagate runtime types to strongly connected components: Time spent was 00:00:00.013
INFO: Variable Type Analysis #2: Time spent was 00:00:00.027
INFO: Create runtime call graph: Time spent was 00:00:00.110
INFO: Load config: Starting
INFO: Load config: Time spent was 00:00:00.170
INFO: Compute entry points: Starting
INFO: Compute entry points: Time spent was 00:00:00.083
INFO: No entry points found.
INFO: python security sensor: Time spent was 00:00:00.712
INFO: python security sensor: Begin: 2024-03-15T18:07:12.637805508Z, End: 2024-03-15T18:07:13.350265842Z, Duration: 00:00:00.712
  Load type hierarchy and UCFGs: Begin: 2024-03-15T18:07:12.638012933Z, End: 2024-03-15T18:07:12.982029318Z, Duration: 00:00:00.344
    Load type hierarchy: Begin: 2024-03-15T18:07:12.638039092Z, End: 2024-03-15T18:07:12.717301392Z, Duration: 00:00:00.079
    Load UCFGs: Begin: 2024-03-15T18:07:12.717491027Z, End: 2024-03-15T18:07:12.981751398Z, Duration: 00:00:00.264
  Check cache: Begin: 2024-03-15T18:07:12.983001548Z, End: 2024-03-15T18:07:12.983659079Z, Duration: 00:00:00.000
    Load cache: Begin: 2024-03-15T18:07:12.983041946Z, End: 2024-03-15T18:07:12.983093144Z, Duration: 00:00:00.000
  Create runtime call graph: Begin: 2024-03-15T18:07:12.983789729Z, End: 2024-03-15T18:07:13.094363133Z, Duration: 00:00:00.110
    Variable Type Analysis #1: Begin: 2024-03-15T18:07:12.984605789Z, End: 2024-03-15T18:07:13.045576320Z, Duration: 00:00:00.060
      Create runtime type propagation graph: Begin: 2024-03-15T18:07:12.985669864Z, End: 2024-03-15T18:07:13.006395949Z, Duration: 00:00:00.020
      Run SCC (Tarjan) on 2168 nodes: Begin: 2024-03-15T18:07:13.007018730Z, End: 2024-03-15T18:07:13.027595077Z, Duration: 00:00:00.020
      Propagate runtime types to strongly connected components: Begin: 2024-03-15T18:07:13.028283390Z, End: 2024-03-15T18:07:13.045279087Z, Duration: 00:00:00.016
    Variable Type Analysis #2: Begin: 2024-03-15T18:07:13.065027011Z, End: 2024-03-15T18:07:13.092319571Z, Duration: 00:00:00.027
      Create runtime type propagation graph: Begin: 2024-03-15T18:07:13.065196836Z, End: 2024-03-15T18:07:13.073605924Z, Duration: 00:00:00.008
      Run SCC (Tarjan) on 2166 nodes: Begin: 2024-03-15T18:07:13.073870807Z, End: 2024-03-15T18:07:13.078024403Z, Duration: 00:00:00.004
      Propagate runtime types to strongly connected components: Begin: 2024-03-15T18:07:13.078340182Z, End: 2024-03-15T18:07:13.092066424Z, Duration: 00:00:00.013
  Load config: Begin: 2024-03-15T18:07:13.094598674Z, End: 2024-03-15T18:07:13.265594384Z, Duration: 00:00:00.170
  Compute entry points: Begin: 2024-03-15T18:07:13.265881613Z, End: 2024-03-15T18:07:13.349672915Z, Duration: 00:00:00.083
INFO: python security sensor peak memory: 174 MB
INFO: Sensor PythonSecuritySensor [security] (done) | time=718ms
INFO: Sensor JsSecuritySensor [security]
INFO: Enabled taint analysis rules: S6350, S2631, S5147, S5696, S6287, S5334, S6105, S5883, S5131, S5144, S2083, S3649, S2076, S6096, S5146
INFO: Load type hierarchy and UCFGs: Starting
INFO: Load type hierarchy: Starting
INFO: Reading type hierarchy from: /workspaces/sage-monorepo/apps/openchallenges/edam-etl/.scannerwork/ucfg2/js
INFO: Read 0 type definitions
INFO: Load type hierarchy: Time spent was 00:00:00.000
INFO: Load UCFGs: Starting
INFO: Load UCFGs: Time spent was 00:00:00.000
INFO: Load type hierarchy and UCFGs: Time spent was 00:00:00.000
INFO: No UCFGs have been included for analysis.
INFO: js security sensor: Time spent was 00:00:00.001
INFO: Sensor JsSecuritySensor [security] (done) | time=3ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=11ms
INFO: SCM Publisher SCM provider for this project is: git
INFO: SCM Publisher 3 source files to be analyzed
INFO: SCM Publisher 2/3 source files have been analyzed (done) | time=308ms
WARN: Missing blame information for the following files:
WARN:   * project.json
WARN: This may lead to missing/broken features in SonarCloud
INFO: CPD Executor 2 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 0 files
INFO: CPD Executor CPD calculation finished (done) | time=0ms
INFO: Analysis report generated in 324ms, dir size=241 KB
INFO: Analysis report compressed in 24ms, zip size=50 KB
INFO: Analysis report uploaded in 516ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=openchallenges-edam-etl
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AY5DS_Ifn0REOZSzzMNf
INFO: Sensor cache published successfully
INFO: Analysis total time: 17.971 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 24.462s
INFO: Final Memory: 27M/120M
INFO: ------------------------------------------------------------------------

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target sonar for project openchallenges-edam-etl (25s)

   View logs and investigate cache misses at https://cloud.nx.app/runs/3bvAMASYW5