Closed pritesha closed 6 years ago
Hi @pritesha
The script of our docker image is already defining the sonar.gitlab.commit_sha
from the CI_BUILD_REF environment variable set by gitlab on any ci pipe. As you can see here. So you don't need to override this variable.
It seems you hit the gitlab.com server in a downtime moment maybe? Yesterday they had an outage.
Anyway I just realized the Gitlab is deprecating some of the environment variables we use in our scanner run script. So we'll update our image fixing this variables (https://docs.gitlab.com/ee/ci/variables/#9-0-renaming). Anyway the old ones should still work.
Hi @gtrias ,
I tried changing this and added exact project id in my properties file, now the process is running fine, but still no comments in the commit ( although I could see in SonarQube dashboard ) any others pointers you have?
Please, paste your current output. BTW, as described in #23 there are some times where the issues mode does not highlight expected code to be highlighted.
This is the current log
Running with gitlab-runner 10.7.0-rc1 (a4699306)
on docker-auto-scale fa6cab46
Using Docker executor with image ciricihq/gitlab-sonar-scanner ...
Pulling docker image ciricihq/gitlab-sonar-scanner ...
Using docker image sha256:770b0d9119fdb66c2522ee91b12dafa6348966fe653e3727dc3ef74066e78f1a for ciricihq/gitlab-sonar-scanner ...
Running on runner-fa6cab46-project-11223344-concurrent-0 via runner-fa6cab46-srm-1524093207-c73cf3d7...
Cloning repository...
Cloning into '/builds/xyz/pqr/sandbox'...
Checking out eea7ea70 as test...
Skipping Git submodules setup
Downloading artifacts for Yarn UI (63774065)...
Downloading artifacts from coordinator... ok id=63774065 responseStatus=200 OK token=xQ-7XJYG
$ gitlab-sonar-scanner
INFO: Scanner configuration file: /sonar-scanner-3.0.3.778/conf/sonar-scanner.properties
INFO: Project root configuration file: /builds/xyz/pqr/sandbox/sonar-project.properties
INFO: SonarQube Scanner 3.0.3.778
INFO: Java 1.8.0_151 Oracle Corporation (64-bit)
INFO: Linux 4.13.5-coreos-r2 amd64
INFO: User cache: /root/.sonar/cache
INFO: Load global settings
INFO: Load global settings (done) | time=506ms
INFO: User cache: /root/.sonar/cache
INFO: Exclude plugins: devcockpit, ldap, authgithub, authbitbucket, pdfreport, authaad, googleanalytics, governance
INFO: Load plugins index
INFO: Load plugins index (done) | time=208ms
INFO: Download sonar-csharp-plugin-5.10.1.1411.jar
INFO: Download sonar-python-plugin-1.8.0.1496.jar
INFO: Download sonar-java-plugin-4.12.0.11033.jar
INFO: Download sonar-scm-git-plugin-1.2.jar
INFO: Download sonar-xml-plugin-1.4.3.1027.jar
INFO: Download sonar-flex-plugin-2.3.jar
INFO: Download sonar-php-plugin-2.10.0.2087.jar
INFO: Download sonar-gitlab-plugin-3.0.0.jar
INFO: Download sonar-scm-svn-plugin-1.5.0.715.jar
INFO: Download sonar-javascript-plugin-3.1.1.5128.jar
INFO: SonarQube server 6.5.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=303ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=1ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=241ms
INFO: Load active rules
INFO: Load active rules (done) | time=1960ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=282ms
INFO: Issues mode
INFO: Scanning only changed files
INFO: Project key: test:test
INFO: ------------- Scan test:test
INFO: Load server rules
INFO: Load server rules (done) | time=236ms
INFO: Base dir: /builds/xyz/pqr/sandbox
INFO: Working dir: /builds/xyz/pqr/sandbox/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: ui/node_modules/**
INFO: 42 files indexed
INFO: 30689 files ignored because of inclusion/exclusion patterns
INFO: Quality profile for cs: Sonar way
INFO: Quality profile for js: Sonar way
INFO: Sensor SonarJavaXmlFileSensor [java]
INFO: Sensor SonarJavaXmlFileSensor [java] (done) | time=3ms
INFO: Sensor Analyzer for "php.ini" files [php]
INFO: Sensor Analyzer for "php.ini" files [php] (done) | time=3ms
INFO: Sensor JavaScript Squid Sensor [javascript]
INFO: 1 source files to be analyzed
INFO: 1/1 source files have been analyzed
INFO: Unit Test Coverage Sensor is started
INFO: Integration Test Coverage Sensor is started
INFO: Overall Coverage Sensor is started
INFO: Sensor JavaScript Squid Sensor [javascript] (done) | time=728ms
INFO: Load server issues
INFO: Load server issues (done) | time=240ms
INFO: Performing issue tracking
WARN: Invalid character encountered in file /builds/kmec/spectrum/sandbox/ui/src/favicon.ico at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
INFO: 54/54 components tracked
INFO: ANALYSIS SUCCESSFUL
INFO: Task total time: 12.435 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 25.794s
INFO: Final Memory: 47M/114M
INFO: ------------------------------------------------------------------------
Job succeeded
@pritesha Are you sure your commit is introducing new issues?
When our sonar CI routine has new issues they're shown in logs as it follows:
......
INFO: Sensor Analyzer for "php.ini" files (done) | time=1ms
INFO: Load server issues
INFO: Load server issues (done) | time=1333ms
INFO: Performing issue tracking
INFO: 483/483 components tracked
WARN: Console report is deprecated. Use SonarLint CLI to have local reports of issues
INFO:
------------- Issues Report -------------
+1 issue
+1 info
-------------------------------------------
INFO: ANALYSIS SUCCESSFUL
INFO: Executing post-job GitLab Commit Issue Publisher
INFO: Report status=success, desc=SonarQube reported 1 issue, with 1 info
INFO: Task total time: 8.435 s
....
Note that sonarqube only reports new issues from the commit which triggered the CI
Hi @gtrias ,
It does introduce new issues, but not publishing them to gitlab , see the latest status in sonarQube for the latest run.
Please @pritesha, kindly paste your .gitlab-ci.yml
file (excluding sensitive parts).
Hi @elboletaire , Please see below
variables:
API_NAME: xyz.abc.Web.API
DEPLOYMENT_NAME: $CI_PROJECT_NAME-$CI_COMMIT_REF_NAME
UI_IMAGE_TAG: $CI_REGISTRY_IMAGE/ui:$CI_COMMIT_REF_NAME
UI_REVIEW_DOMAIN: $CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG.dev.myorg.com
UI_PROD_DOMAIN: $CI_PROJECT_NAME.myorg.com
API_IMAGE_TAG: $CI_REGISTRY_IMAGE/api:$CI_COMMIT_REF_NAME
API_REVIEW_DOMAIN: $CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG-api.dev.myorg.com
API_PROD_DOMAIN: $CI_PROJECT_NAME-api.myorg.com
stages:
- Init
- Prebuild
- Build
- Package
- Deploy
Yarn UI:
stage: Init
image: registry.gitlab.com/xyz/abc/docker/node:latest
script:
- cd ui
- yarn cache clean
- yarn
artifacts:
expire_in: 1 hour
paths:
- ./ui/node_modules
only:
- branches
Run UI Linter:
stage: Prebuild
image: registry.gitlab.com/xyz/abc/docker/node:latest
dependencies:
- Yarn UI
script:
- cd ui
- npm run lint
only:
- branches
except:
- master
Run SonarQube:
stage: Prebuild
image: ciricihq/gitlab-sonar-scanner
variables:
SONAR_URL: http://sonar.myorg.com:9000
SONAR_ANALYSIS_MODE: issues
script:
- gitlab-sonar-scanner
only:
- branches
except:
- master
Run SonarQube-reports:
stage: Prebuild
image: ciricihq/gitlab-sonar-scanner
variables:
SONAR_URL: http://sonar.myorg.com:9000
SONAR_ANALYSIS_MODE: publish
script:
- gitlab-sonar-scanner
only:
- branches
except:
- master
Run UI Tests:
stage: Prebuild
image: registry.gitlab.com/xyz/abc/docker/node-chrome:latest
dependencies:
- Yarn UI
script:
- cd ui
- echo "TODO"
only:
- branches
except:
- master
Run API Tests:
stage: Prebuild
image: microsoft/aspnetcore-build:2.0
variables:
PROJECT_PATH: $API_NAME.Tests.Unit/$API_NAME.Tests.Unit.csproj
script:
- echo ">> project=$PROJECT_PATH"
- cd api
- dotnet test ./$PROJECT_PATH
only:
- branches
except:
- master
Build UI Review:
stage: Build
image: registry.gitlab.com/xyz/abc/docker/node:latest
dependencies:
- Yarn UI
script:
- cd ui
- echo ">> url=$API_REVIEW_DOMAIN"
- export production=true apiUrl=http://$API_REVIEW_DOMAIN
- npm run build
artifacts:
paths:
- ./ui/dist
only:
- branches
except:
- master
Build UI Production:
stage: Build
image: registry.gitlab.com/xyz/abc/docker/node:latest
dependencies:
- Yarn UI
script:
- cd ui
- echo ">> url=$API_PROD_DOMAIN"
- export production=true apiUrl=http://$API_PROD_DOMAIN
- npm run build
artifacts:
paths:
- ./ui/dist
only:
- master
Package UI Review:
stage: Package
image: docker:latest
services:
- docker:dind
dependencies:
- Build UI Review
variables:
DOCKERFILE_PATH: ui/Dockerfile
DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/ui
script:
- echo ">> file=$DOCKERFILE_PATH"
- echo ">> image=$DOCKER_IMAGE_PATH"
- echo ">> tag=$UI_IMAGE_TAG"
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $UI_IMAGE_TAG -f ./$DOCKERFILE_PATH ./ui
- docker push $DOCKER_IMAGE_PATH
only:
- branches
except:
- master
Package UI Production:
stage: Package
image: docker:latest
services:
- docker:dind
dependencies:
- Build UI Production
variables:
DOCKERFILE_PATH: ui/Dockerfile
DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/ui
script:
- echo ">> file=$DOCKERFILE_PATH"
- echo ">> image=$DOCKER_IMAGE_PATH"
- echo ">> tag=$UI_IMAGE_TAG"
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $UI_IMAGE_TAG -f ./$DOCKERFILE_PATH ./ui
- docker push $DOCKER_IMAGE_PATH
only:
- master
Package API Review:
stage: Package
image: docker:latest
services:
- docker:dind
variables:
DOCKERFILE_PATH: api/$API_NAME/Dockerfile
DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/api
script:
- echo ">> file=$DOCKERFILE_PATH"
- echo ">> image=$DOCKER_IMAGE_PATH"
- echo ">> tag=$API_IMAGE_TAG"
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $API_IMAGE_TAG -f ./$DOCKERFILE_PATH ./api
- docker push $DOCKER_IMAGE_PATH
only:
- branches
except:
- master
Package API Production:
stage: Package
image: docker:latest
services:
- docker:dind
variables:
DOCKERFILE_PATH: api/$API_NAME/Dockerfile
DOCKER_IMAGE_PATH: $CI_REGISTRY_IMAGE/api
script:
- echo ">> file=$DOCKERFILE_PATH"
- echo ">> image=$DOCKER_IMAGE_PATH"
- echo ">> tag=$API_IMAGE_TAG"
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $API_IMAGE_TAG -f ./$DOCKERFILE_PATH ./api
- docker push $DOCKER_IMAGE_PATH
only:
- master
Deploy Review:
stage: Deploy
image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
dependencies:
- Package UI Review
- Package API Review
script:
- kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
- kubectl config use-context $CI_PROJECT_NAME
- echo ">> registry=$CI_REGISTRY"
- echo ">> name=$DEPLOYMENT_NAME"
- echo ">> ui=$UI_IMAGE_TAG --> $UI_REVIEW_DOMAIN"
- echo ">> api=$API_IMAGE_TAG --> $API_REVIEW_DOMAIN"
- echo ">> sha=$CI_COMMIT_SHA"
- sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_REVIEW_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_REVIEW_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
- kubectl apply -f ./deployment.yaml
environment:
name: review/$CI_COMMIT_REF_NAME
url: http://$CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG.dev.myorg.com
on_stop: Teardown Review
only:
- branches
except:
- master
Teardown Review:
stage: Deploy
image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
script:
- kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
- kubectl config use-context $CI_PROJECT_NAME
- echo ">> registry=$CI_REGISTRY"
- echo ">> name=$DEPLOYMENT_NAME"
- echo ">> ui=$UI_IMAGE_TAG --> $UI_REVIEW_DOMAIN"
- echo ">> api=$API_IMAGE_TAG --> $API_REVIEW_DOMAIN"
- echo ">> sha=$CI_COMMIT_SHA"
- sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_REVIEW_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_REVIEW_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
- kubectl delete -f ./deployment.yaml
when: manual
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
only:
- branches
except:
- master
Deploy Production:
stage: Deploy
image: registry.gitlab.com/xyz/abc/docker/kubectl:latest
dependencies:
- Package UI Production
- Package API Production
script:
- kubectl config set-cluster $CI_PROJECT_NAME --server=$KUBE_URL --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-credentials $CI_PROJECT_NAME --token=$KUBE_TOKEN --certificate-authority=$KUBE_CA_PEM_FILE
- kubectl config set-context $CI_PROJECT_NAME --cluster=$CI_PROJECT_NAME --user=$CI_PROJECT_NAME
- kubectl config use-context $CI_PROJECT_NAME
- echo ">> registry=$CI_REGISTRY"
- echo ">> name=$DEPLOYMENT_NAME"
- echo ">> ui=$UI_IMAGE_TAG --> $UI_PROD_DOMAIN"
- echo ">> api=$API_IMAGE_TAG --> $API_PROD_DOMAIN"
- echo ">> sha=$CI_COMMIT_SHA"
- sed -i "s~{DEPLOYMENT_REGISTRY}~$CI_REGISTRY~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_NAME}~$DEPLOYMENT_NAME~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_UI}~$UI_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_IMAGE_API}~$API_IMAGE_TAG~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_UI}~$UI_PROD_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_DOMAIN_API}~$API_PROD_DOMAIN~g" ./deployment.yaml
- sed -i "s~{DEPLOYMENT_SHA}~$CI_COMMIT_SHA~g" ./deployment.yaml
- kubectl apply -f ./deployment.yaml
environment:
name: production
url: http://$CI_PROJECT_NAME.myorg.com
only:
- master
@pritesha we don't see anything weird in your configuration. From our side, apparently it should work.
Have you tried to run the sonarqube scanner without our container? Seems like you have an issue with gitlab-sonar plugin or with sonar itself.
Closing due to inactivity. Feel free to open it again if you thing this is an error, or if you have more information.
BTW, you should try to run the sonarqube scanner without the container to properly determine where the error comes from.
How can we run the sonarqube scanner without the container?
Thanks for the help.
Please, read the official sonarqube documentation about it: https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner
Hi, I am using gitlab-sonar-plugin to run sonar scanner for each branch on every new commit. I am using gabrie-allaigre/sonar-gitlab-plugin for sonnar-gitlab integration.
My expectation is, if I run sonar scanner is issue mode, it should add a comment in my commit with issues and with publishing mode it should publish the report to sonar server.
So, till now I can publish my report to sonar server with all issue and git lab code for that particular commit, but not been able to see any comment on my gitlab commit.
Then, I realised that I need to pass the commit sha, and changed this command to gitlab cli
script:
post this change, my sonar scanner is failing constantly with below exception ERROR: Error during SonarQube Scanner execution ERROR: Failed to execute project builder: com.talanlabs.sonar.plugins.gitlab.CommitProjectBuilder ERROR: Caused by: Unable to perform GitLab WS operation ERROR: Caused by: GitLab is not responding ERROR: Caused by: Server returned HTTP response code: 502 for URL: https://gitlab.com/api/v4/projects?archived=false&membership=false&order_by=created_at&owned=false&page=57&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false ERROR: ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.
I am using
gabrie-allaigre/sonar-gitlab-plugin version 3.0.0
image: ciricihq/gitlab-sonar-scanner
Gitlab version 10.7.0-rc5-ee
sonnarQube version 6.5.0.27846
My gitlab yml task file looks like this
Run SonarQube:
stage: Prebuild
image: ciricihq/gitlab-sonar-scanner
variables:
SONAR_URL: http://sonar.my.sonar.url
SONAR_ANALYSIS_MODE: issues
script:
- gitlab-sonar-scanner -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHA
only:
- branches
except:
- master
and my sonar-project.properties files is as below
sonar.projectKey=test:test sonar.exclusions=ui/node_modules/** sonar.sources=. sonar.gitlab.project_id=git@gitlab.com:xyz/pqr/abc.git