search

ciricihq / gitlab-sonar-scanner

This project is no longer maintained
GNU General Public License v3.0
80 stars 62 forks source link

sonar scanner failed with exception GitLab is not responding #24

Closed pritesha closed 6 years ago

pritesha commented 6 years ago

Hi, I am using gitlab-sonar-plugin to run sonar scanner for each branch on every new commit. I am using gabrie-allaigre/sonar-gitlab-plugin for sonnar-gitlab integration.

My expectation is, if I run sonar scanner is issue mode, it should add a comment in my commit with issues and with publishing mode it should publish the report to sonar server.

So, till now I can publish my report to sonar server with all issue and git lab code for that particular commit, but not been able to see any comment on my gitlab commit.

Then, I realised that I need to pass the commit sha, and changed this command to gitlab cli

script:

post this change, my sonar scanner is failing constantly with below exception ERROR: Error during SonarQube Scanner execution ERROR: Failed to execute project builder: com.talanlabs.sonar.plugins.gitlab.CommitProjectBuilder ERROR: Caused by: Unable to perform GitLab WS operation ERROR: Caused by: GitLab is not responding ERROR: Caused by: Server returned HTTP response code: 502 for URL: https://gitlab.com/api/v4/projects?archived=false&membership=false&order_by=created_at&owned=false&page=57&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false ERROR: ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

I am using

My gitlab yml task file looks like this Run SonarQube: stage: Prebuild image: ciricihq/gitlab-sonar-scanner variables: SONAR_URL: http://sonar.my.sonar.url SONAR_ANALYSIS_MODE: issues script: - gitlab-sonar-scanner -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHA only: - branches except: - master

and my sonar-project.properties files is as below sonar.projectKey=test:test sonar.exclusions=ui/node_modules/** sonar.sources=. sonar.gitlab.project_id=git@gitlab.com:xyz/pqr/abc.git

gtrias commented 6 years ago

Hi @pritesha

The script of our docker image is already defining the sonar.gitlab.commit_sha from the CI_BUILD_REF environment variable set by gitlab on any ci pipe. As you can see here. So you don't need to override this variable.

It seems you hit the gitlab.com server in a downtime moment maybe? Yesterday they had an outage.

Anyway I just realized the Gitlab is deprecating some of the environment variables we use in our scanner run script. So we'll update our image fixing this variables (https://docs.gitlab.com/ee/ci/variables/#9-0-renaming). Anyway the old ones should still work.

pritesha commented 6 years ago

Hi @gtrias ,

I tried changing this and added exact project id in my properties file, now the process is running fine, but still no comments in the commit ( although I could see in SonarQube dashboard ) any others pointers you have?

elboletaire commented 6 years ago

Please, paste your current output. BTW, as described in #23 there are some times where the issues mode does not highlight expected code to be highlighted.

pritesha commented 6 years ago

This is the current log 

Running with gitlab-runner 10.7.0-rc1 (a4699306)
  on docker-auto-scale fa6cab46
Using Docker executor with image ciricihq/gitlab-sonar-scanner ...
Pulling docker image ciricihq/gitlab-sonar-scanner ...
Using docker image sha256:770b0d9119fdb66c2522ee91b12dafa6348966fe653e3727dc3ef74066e78f1a for ciricihq/gitlab-sonar-scanner ...
Running on runner-fa6cab46-project-11223344-concurrent-0 via runner-fa6cab46-srm-1524093207-c73cf3d7...
Cloning repository...
Cloning into '/builds/xyz/pqr/sandbox'...
Checking out eea7ea70 as test...
Skipping Git submodules setup
Downloading artifacts for Yarn UI (63774065)...
Downloading artifacts from coordinator... ok        id=63774065 responseStatus=200 OK token=xQ-7XJYG
$ gitlab-sonar-scanner
INFO: Scanner configuration file: /sonar-scanner-3.0.3.778/conf/sonar-scanner.properties
INFO: Project root configuration file: /builds/xyz/pqr/sandbox/sonar-project.properties
INFO: SonarQube Scanner 3.0.3.778
INFO: Java 1.8.0_151 Oracle Corporation (64-bit)
INFO: Linux 4.13.5-coreos-r2 amd64
INFO: User cache: /root/.sonar/cache
INFO: Load global settings
INFO: Load global settings (done) | time=506ms
INFO: User cache: /root/.sonar/cache
INFO: Exclude plugins: devcockpit, ldap, authgithub, authbitbucket, pdfreport, authaad, googleanalytics, governance
INFO: Load plugins index
INFO: Load plugins index (done) | time=208ms
INFO: Download sonar-csharp-plugin-5.10.1.1411.jar
INFO: Download sonar-python-plugin-1.8.0.1496.jar
INFO: Download sonar-java-plugin-4.12.0.11033.jar
INFO: Download sonar-scm-git-plugin-1.2.jar
INFO: Download sonar-xml-plugin-1.4.3.1027.jar
INFO: Download sonar-flex-plugin-2.3.jar
INFO: Download sonar-php-plugin-2.10.0.2087.jar
INFO: Download sonar-gitlab-plugin-3.0.0.jar
INFO: Download sonar-scm-svn-plugin-1.5.0.715.jar
INFO: Download sonar-javascript-plugin-3.1.1.5128.jar
INFO: SonarQube server 6.5.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=303ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=1ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=241ms
INFO: Load active rules
INFO: Load active rules (done) | time=1960ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=282ms
INFO: Issues mode
INFO: Scanning only changed files
INFO: Project key: test:test
INFO: -------------  Scan test:test
INFO: Load server rules
INFO: Load server rules (done) | time=236ms
INFO: Base dir: /builds/xyz/pqr/sandbox
INFO: Working dir: /builds/xyz/pqr/sandbox/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources: 
INFO:   ui/node_modules/**
INFO: 42 files indexed
INFO: 30689 files ignored because of inclusion/exclusion patterns
INFO: Quality profile for cs: Sonar way
INFO: Quality profile for js: Sonar way
INFO: Sensor SonarJavaXmlFileSensor [java]
INFO: Sensor SonarJavaXmlFileSensor [java] (done) | time=3ms
INFO: Sensor Analyzer for "php.ini" files [php]
INFO: Sensor Analyzer for "php.ini" files [php] (done) | time=3ms
INFO: Sensor JavaScript Squid Sensor [javascript]
INFO: 1 source files to be analyzed
INFO: 1/1 source files have been analyzed
INFO: Unit Test Coverage Sensor is started
INFO: Integration Test Coverage Sensor is started
INFO: Overall Coverage Sensor is started
INFO: Sensor JavaScript Squid Sensor [javascript] (done) | time=728ms
INFO: Load server issues
INFO: Load server issues (done) | time=240ms
INFO: Performing issue tracking
WARN: Invalid character encountered in file /builds/kmec/spectrum/sandbox/ui/src/favicon.ico at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
INFO: 54/54 components tracked
INFO: ANALYSIS SUCCESSFUL
INFO: Task total time: 12.435 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 25.794s
INFO: Final Memory: 47M/114M
INFO: ------------------------------------------------------------------------
Job succeeded
gtrias commented 6 years ago

@pritesha Are you sure your commit is introducing new issues?

When our sonar CI routine has new issues they're shown in logs as it follows:

......
INFO: Sensor Analyzer for "php.ini" files (done) | time=1ms
INFO: Load server issues
INFO: Load server issues (done) | time=1333ms
INFO: Performing issue tracking
INFO: 483/483 components tracked
WARN: Console report is deprecated. Use SonarLint CLI to have local reports of issues
INFO: 

-------------  Issues Report  -------------

        +1 issue

        +1 info

-------------------------------------------

INFO: ANALYSIS SUCCESSFUL
INFO: Executing post-job GitLab Commit Issue Publisher
INFO: Report status=success, desc=SonarQube reported 1 issue, with 1 info
INFO: Task total time: 8.435 s
....

Note that sonarqube only reports new issues from the commit which triggered the CI

pritesha commented 6 years ago

Hi @gtrias , It does introduce new issues, but not publishing them to gitlab , see the latest status in sonarQube for the latest run. image

elboletaire commented 6 years ago

Please @pritesha, kindly paste your .gitlab-ci.yml file (excluding sensitive parts).

pritesha commented 6 years ago

Hi @elboletaire , Please see below 

variables:
  API_NAME: xyz.abc.Web.API
  DEPLOYMENT_NAME: $CI_PROJECT_NAME-$CI_COMMIT_REF_NAME
  UI_IMAGE_TAG: $CI_REGISTRY_IMAGE/ui:$CI_COMMIT_REF_NAME
  UI_REVIEW_DOMAIN: $CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG.dev.myorg.com
  UI_PROD_DOMAIN: $CI_PROJECT_NAME.myorg.com
  API_IMAGE_TAG: $CI_REGISTRY_IMAGE/api:$CI_COMMIT_REF_NAME
  API_REVIEW_DOMAIN: $CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG-api.dev.myorg.com
  API_PROD_DOMAIN: $CI_PROJECT_NAME-api.myorg.com

stages:
  - Init 
  - Prebuild
  - Build
  - Package
  - Deploy

Yarn UI:
  stage: Init
  image: registry.gitlab.com/xyz/abc/docker/node:latest
  script:
    - cd ui
    - yarn cache clean
    - yarn
  artifacts:
    expire_in: 1 hour
    paths:
      - ./ui/node_modules
  only:
    - branches

Run UI Linter:
  stage: Prebuild
  image: registry.gitlab.com/xyz/abc/docker/node:latest
  dependencies:
    - Yarn UI
  script:
    - cd ui
    - npm run lint
  only:
    - branches
  except:
    - master

Run SonarQube:
  stage: Prebuild
  image: ciricihq/gitlab-sonar-scanner
  variables:
    SONAR_URL: http://sonar.myorg.com:9000
    SONAR_ANALYSIS_MODE: issues
  script:
    - gitlab-sonar-scanner
  only:
    - branches
  except:
    - master

Run SonarQube-reports:
  stage: Prebuild
  image: ciricihq/gitlab-sonar-scanner
  variables:
    SONAR_URL: http://sonar.myorg.com:9000
    SONAR_ANALYSIS_MODE: publish
  script:
    - gitlab-sonar-scanner
  only:
    - branches
  except:
    - master

Run UI Tests:
  stage: Prebuild
  image: registry.gitlab.com/xyz/abc/docker/node-chrome:latest
  dependencies:
    - Yarn UI
  script:
    - cd ui
    - echo "TODO"
  only:
    - branches
  except:
    - master

Run API Tests:
  stage: Prebuild
  image: microsoft/aspnetcore-build:2.0
  variables:
    PROJECT_PATH: $API_NAME.Tests.Unit/$API_NAME.Tests.Unit.csproj
  script:
    - echo ">> project=$PROJECT_PATH"
    - cd api
    - dotnet test ./$PROJECT_PATH
  only:
    - branches
  except:
    - master

Build UI Review:
  stage: Build
  image: registry.gitlab.com/xyz/abc/docker/node:latest
  dependencies:
    - Yarn UI
  script:
    - cd ui
    - echo ">> url=$API_REVIEW_DOMAIN"
    - export production=true apiUrl=http://$API_REVIEW_DOMAIN
    - npm run build
  artifacts:
    paths:
      - ./ui/dist
  only:
    - branches
  except:
    - master

Build UI Production:
  stage: Build
  image: registry.gitlab.com/xyz/abc/docker/node:latest
  dependencies:
    - Yarn UI
  script:
    - cd ui
    - echo ">> url=$API_PROD_DOMAIN"
    - export production=true apiUrl=http://$API_PROD_DOMAIN
    - npm run build
  artifacts:
    paths:
      - ./ui/dist
  only:
    - master

Package UI Review:
  stage: Package
  image: docker:latest
  services:
    - docker:dind
  dependencies:
    - Build UI Review
  variables:
    DOCKERFILE_PATH: ui/Dockerfile
    DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/ui
  script:
    - echo ">> file=$DOCKERFILE_PATH"
    - echo ">> image=$DOCKER_IMAGE_PATH"
    - echo ">> tag=$UI_IMAGE_TAG"
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $UI_IMAGE_TAG -f ./$DOCKERFILE_PATH ./ui
    - docker push $DOCKER_IMAGE_PATH
  only:
    - branches
  except:
    - master

Package UI Production:
  stage: Package
  image: docker:latest
  services:
    - docker:dind
  dependencies:
    - Build UI Production
  variables:
    DOCKERFILE_PATH: ui/Dockerfile
    DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/ui
  script:
    - echo ">> file=$DOCKERFILE_PATH"
    - echo ">> image=$DOCKER_IMAGE_PATH"
    - echo ">> tag=$UI_IMAGE_TAG"
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $UI_IMAGE_TAG -f ./$DOCKERFILE_PATH ./ui
    - docker push $DOCKER_IMAGE_PATH
  only:
    - master

Package API Review:
  stage: Package
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKERFILE_PATH: api/$API_NAME/Dockerfile
    DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/api
  script:
    - echo ">> file=$DOCKERFILE_PATH"
    - echo ">> image=$DOCKER_IMAGE_PATH"
    - echo ">> tag=$API_IMAGE_TAG"
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $API_IMAGE_TAG -f ./$DOCKERFILE_PATH ./api
    - docker push $DOCKER_IMAGE_PATH
  only:
    - branches
  except:
    - master

Package API Production:
  stage: Package
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKERFILE_PATH: api/$API_NAME/Dockerfile
    DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/api
  script:
    - echo ">> file=$DOCKERFILE_PATH"
    - echo ">> image=$DOCKER_IMAGE_PATH"
    - echo ">> tag=$API_IMAGE_TAG"
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $API_IMAGE_TAG -f ./$DOCKERFILE_PATH ./api
    - docker push $DOCKER_IMAGE_PATH
  only:
    - master

Deploy Review:
  stage: Deploy
  image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
  dependencies:
    - Package UI Review
    - Package API Review
  script:  
    - kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
    - kubectl config use-context $CI_PROJECT_NAME
    - echo ">> registry=$CI_REGISTRY"
    - echo ">> name=$DEPLOYMENT_NAME"
    - echo ">> ui=$UI_IMAGE_TAG --> $UI_REVIEW_DOMAIN"
    - echo ">> api=$API_IMAGE_TAG --> $API_REVIEW_DOMAIN"
    - echo ">> sha=$CI_COMMIT_SHA"
    - sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_REVIEW_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_REVIEW_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
    - kubectl apply -f ./deployment.yaml
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: http://$CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG.dev.myorg.com
    on_stop: Teardown Review
  only:
    - branches
  except:
    - master

Teardown Review:
  stage: Deploy
  image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
  script:  
    - kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
    - kubectl config use-context $CI_PROJECT_NAME
    - echo ">> registry=$CI_REGISTRY"
    - echo ">> name=$DEPLOYMENT_NAME"
    - echo ">> ui=$UI_IMAGE_TAG --> $UI_REVIEW_DOMAIN"
    - echo ">> api=$API_IMAGE_TAG --> $API_REVIEW_DOMAIN"
    - echo ">> sha=$CI_COMMIT_SHA"
    - sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_REVIEW_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_REVIEW_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
    - kubectl delete -f ./deployment.yaml
  when: manual
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop
  only:
    - branches
  except:
    - master

Deploy Production:
  stage: Deploy
  image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
  dependencies:
    - Package UI Production
    - Package API Production
  script:  
    - kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
    - kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
    - kubectl config use-context $CI_PROJECT_NAME
    - echo ">> registry=$CI_REGISTRY"
    - echo ">> name=$DEPLOYMENT_NAME"
    - echo ">> ui=$UI_IMAGE_TAG --> $UI_PROD_DOMAIN"
    - echo ">> api=$API_IMAGE_TAG --> $API_PROD_DOMAIN"
    - echo ">> sha=$CI_COMMIT_SHA"
    - sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_PROD_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_PROD_DOMAIN~g" ./deployment.yaml
    - sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
    - kubectl apply -f ./deployment.yaml
  environment:
    name: production
    url: http://$CI_PROJECT_NAME.myorg.com
  only:
    - master
elboletaire commented 6 years ago

@pritesha we don't see anything weird in your configuration. From our side, apparently it should work.

Have you tried to run the sonarqube scanner without our container? Seems like you have an issue with gitlab-sonar plugin or with sonar itself.

elboletaire commented 6 years ago

Closing due to inactivity. Feel free to open it again if you thing this is an error, or if you have more information.

BTW, you should try to run the sonarqube scanner without the container to properly determine where the error comes from.

MuyBien commented 6 years ago

How can we run the sonarqube scanner without the container?

Thanks for the help.

elboletaire commented 6 years ago

Please, read the official sonarqube documentation about it: https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner