search

dennys / comment-utterances

0 stars 0 forks source link

en/doc/devops/grafana-gitlab-integration/ #2

Open utterances-bot opened 1 year ago

utterances-bot commented 1 year ago

Use Grafana to Manage GitLab CI/CD Pipelines | Dennys Diary

Requirement When there are lots of projects on GitLab, it's not easy to know the overall build status of their CI/CD Pipelines. Use Grafana to Manage GitLab …

https://dennys.github.io/en/doc/devops/grafana-gitlab-integration/

MahmoodKamali commented 1 year ago

I ollowd up your post step by step, however, there was no success. I have no idea what to do.

dennys commented 1 year ago

Please check http://your.prometheus.server/targets, you can see the state column. If it's red, you can see the error message. If it's blue, then you need to check Grafana's configuration.

MahmoodKamali commented 1 year ago

Interestingly, It is blue. The problem is, there is no GITLAB metric on Prometheus. I think Grafna has nothing to do with it. Unfortunately, I cannot attach the files to show you it is so bizarre.

dennys commented 1 year ago

Do you check your GitLab? It should be https://your.gitlab/metrics/ (the URL is based on your configuration, you can check https://docs.gitlab.com/ee/administration/monitoring/prometheus/. If you cannot see it on GitLab... it seems not easy to debug, I'm not sure does GitLab show anything in their log.

MahmoodKamali commented 1 year ago

I have set up the Prometheus server on Server A and GitLab on Server B. Prometheus and Grafana are on Server A. Prometheus is listening to all servers based on the installed Node exporter and now, I need to monitor GitLab-ci jobs monitoring. I changed GitLab.rn file, however, there was no change and still get no metrics on the Prometheus. I', confused. Unfortunately, there is no clear document to be able to read and apply it.

dennys commented 1 year ago

You use solution 1 (GitLab's build-in exporter) or solution 2 (gitlab-ci-pipelines-exporter)?

MahmoodKamali commented 1 year ago

Solution 2. I don't need to enable the built-in Prometheus of GitLab. I need to monitor CI/CD jobs.

dennys commented 1 year ago

If you use solution 2, do you check your exporter? You can check it at http://your.docker:9103/metrics (the port is based on your yml).

MahmoodKamali commented 1 year ago

it seems to work. however, the docker container logs show something does not work properly.

Docker logs:

time="2022-10-12T08:46:00Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:46:30Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:47:00Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:47:30Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:48:00Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:48:30Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:49:00Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:49:30Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:50:00Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:50:30Z" level=info msg="scheduling metrics pull" environments-count=0 refs-count=0 time="2022-10-12T08:50:30Z" level=info msg="scheduling refs from projects pull" projects-count=2 time="2022-10-12T08:50:30Z" level=info msg="starting 'metrics' garbage collection" time="2022-10-12T08:50:30Z" level=info msg="ending 'metrics' garbage collection" time="2022-10-12T08:50:30Z" level=warning msg="pulling refs from project" error="Get \"https://gitlab.xxxx.xxx/api/v4/projects/gitlab-org%2Fgitlab-runner/repository/branches?page=1&per_page=100\": dial tcp: lookup gitlab.xxxx.xxx on 127.0.0.11:53: no such host" project-name=gitlab-org/gitlab-runner time="2022-10-12T08:50:30Z" level=warning msg="pulling refs from project" error="Get \"https://gitlab.xxxx.xxx/api/v4/projects/gitlab-org%2Fcharts%2Fauto-deploy-app/repository/branches?page=1&per_page=100\": dial tcp: lookup gitlab.xxxx.xxx on 127.0.0.11:53: no such host" project-name=gitlab-org/charts/auto-deploy-app

dennys commented 1 year ago

Hi, it seems you delete a reply? I see it in my email box. You should see some metrics start like "gitlab_ci_pineline_xxxx" If you only see metrics like "gcpeprojects***" and "promhttp_metric_handler_errors_total", I think gitlab_ci_pineline_exporter doesn't get data from GitLab.

Do you write a gitlab-ci-pipelines-exporter.yml? If not, you can reference it gitlab-ci-pipelines-exporter.yml

MahmoodKamali commented 1 year ago

Yes, I just wanted to modify my reply. As you know, my main problem is that you commented and don't know what to do. I followed up on the instruction (However it is not so clear) and couldn't set it up correctly. My last reply shows the error that it is not fixed yet. Even more, I mounted the /etc/hosts to help the container to see my Gitlab URL correctly, but, again get the same error.

dennys commented 1 year ago

It seems you run GitLab and gitlab-ci-pipelines-exporter at the same docker? I run them in different VMs. Is it possible to use external IP? For example, your docker's IP is 192.168.1.20, you can expose your GitLab to 192.168.1.20:8000. And gitlab-ci-pipelines-exporter can connect to http://192.168.1.20:8000. This is just a suggestion, my situation is not the same as yours, I cannot reproduce your issue in my environment.

MahmoodKamali commented 1 year ago

I run Prometheus and Grafana containers on a VM with IP 192.168.1.1 (For example). I run a Gitlab server container on a VM with IP 192.168.19.2.

Before running the gitlab-ci-pipelines-exporter container, I ran Node-Exporter and cAdvisor containers on ports 9100 and 8080 on the Gitlab server VM to send their metrics to the Prometheus servers, which work fine and now I can monitor the OS and all running containers. Everything works fine.

Then, I decided to monitor Gitlab-ci jobs, environment, and so on by the Prometheus. So, I ran the gitlab-ci-pipelines-exporter container on port 8081 on the Gitlab server VM. It seems Prometheus can see the exporter and it is on OK status, however, there are no metrics. This is the point I am confused about and don't know what to do. Even, I sent you the log of the gitlab-exporter container that shows there is an error. I think this is the point I couldn't solve.

Unfortunately, here it is not possible to send pics.