How can I visualise Gitlab data?

[pilgrimage80]

I want to visualise the data of the users in Gitlab, such as how many issues they have opened and how many issues they have closed. According to the document below, I fill in the project.json and setup.cfg sections, but it does not work.

https://chaoss.github.io/grimoirelab-tutorial/docs/data-sources/configurations/#gitlab

When I look at the last line of setup.cfg, I think I need to update the following data according to gitlab. How do I do this?

What could I be missing?

[enrich_demography:git]

[enrich_areas_of_code:git]

in_index = git_demo_raw

out_index = git-aoc_demo_enriched

[enrich_onion:git]

in_index = git

out_index = git-onion_demo_enriched

contribs_field = hash

[sduenas]

Can you complete the issue with information about how you are running grimoirelab, the contents of your files, etc?

[pilgrimage80]

I installed everything as docker-compose according to the link below.

https://chaoss.github.io/grimoirelab-tutorial/docs/getting-started/setup/

My Project.json : { "XX": { "gitlab:issue": [ "https://gitlabtest/t/xx.git"

    ]
}

}

setup.cfg [general] short_name = GrimoireLab update = true min_update_delay = 60 debug = false logs_dir = /home/grimoire/logs bulk_size = 100 scroll_size = 100 aliases_file = /home/grimoire/aliases.json

[projects] projects_file = /home/grimoire/conf/projects.json

[es_collection] url = http://elasticsearch:9200

[es_enrichment] url = http://elasticsearch:9200 autorefresh = true

[sortinghat] host = nginx user = root password = root port = 8000 path = /identities/api/ ssl = false database = sortinghat_db autoprofile = [github, git] matching = [email,name,username] sleep_for = 100 unaffiliated_group = Unknown affiliate = true

[panels] kibiter_time_from = now-90d kibiter_default_index = git kibiter_url = http://kibiter:5601 kibiter_version = 6.8.6-3 gitlab-issues = true

[phases] collection = true identities = true enrichment = true panels = true

[gitlab:issues] api-token = kSMNn3hKvxWGXaC-bruQ raw_index = gitlab_issues_demo_raw enriched_index = gitlab_issues_demo_enriched no-archive = true enterprise-url = https://gitlabtest sleep-for-rate = true

[enrich_onion:gitlab-issue] # (optional) in_index = gitlab-issues_enriched out_index = gitlab-issues-onion_enriched data_source = gitlab-issues

When I open localhost:5601 it says no result found and no data is displayed.

There are some patterns in the discover section from the menu, which one of them should be selected

kibana_1 grimoirelab-sigils affliations

What am I doing wrong?

[zhquan]

Hi @pilgrimage80

Could you check if you have indexes and aliases? Go to Dev Tools and run:

GET _cat/indices GET _cat/aliases

And which version of each component are you using?

[pilgrimage80]

GET _cat/indices green open .kibana_1 BhACCJh4QxCfbb3FrHy6WQ 1 0 131 7 279.4kb 279.4kb yellow open .grimoirelab-sigils h9Y2xjjKQWWiYDuHr07VtQ 5 1 22 0 59.6kb 59.6kb

GET _cat/aliases bana .kibana_1 - - -

mariadb 10.6
grimoirelab/sortinghat-worker latest
grimoirelab/sortinghat latest
grimoirelab/grimoirelab latest
docker.elastic.co/kibana/kibana 8.12.2
docker.elastic.co/elasticsearch/elasticsearch 8.12.2
nginx latest
redis latest
bitergia/kibiter community-v6.8.6-3
docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.6

[zhquan]

You have 2 elasticsearch and 2 kibana I guess grimoirelab is connected to ES 6.8.6.

You don't have any index created by grimoirelab. Could you check the grimoirelab logs if there is any error?

[pilgrimage80]

How can I check the logs on grimoirelab?

[pilgrimage80]

Where are the logs?

[zhquan]

If you are using https://github.com/chaoss/grimoirelab/blob/master/docker-compose/docker-compose.yml#L50 you can find it at /tmp/all.log or docker logs <mordred_container_name>

[pilgrimage80]

Hello, I tried to proceed according to the document below without any errors, but I am having a problem as follows.

docker-compose up -d

https://bitergia.com/blog/opensource/grimoirelab-to-measure-organizations-private-development/

After running the command, docker containers are up but after a while the following container is exited. 945f836d644e grimoirelab/grimoirelab:latest "/bin/sh -c ${DEPLOY…" 19 hours ago Exited (1) 12 minutes ago dockercompose_mordred_1

In addition, I chose the Access token of a project that I want to get internal gitlab data in the form of api and read api, and I chose Guest as the Role. Can you confirm the correctness?

As a result, after logging in to http://localhost:5601, I see things like api, gitlab issue as index patterns. Even if I select any of these as default, the dashboard does not give an error but says no result found, as if it does not reach the data within the company.

Where could I have made a mistake? I need your support on this issue.

[pilgrimage80]

My indexes and aliases?

GET _cat/indices yellow open .grimoirelab-sigils _xV71ns4SaSCH5wC2ISLCQ 5 1 22 0 68.6kb 68.6kb green open kibana_sample_data_ecommerce d4rcO8D4Sg-c-lF_jmj4_w 1 0 4675 0 4.8mb 4.8mb green open .kibana_1 wegrmJ1jS7OAiFM2QtnVqA 1 0 149 5 281.8kb 281.8kb yellow open api D0PN4l8BS4aKHPGrFetxLA 5 1 1 0 5.4kb 5.4kb

GET _cat/aliases

.kibana .kibana_1 - - -

[pilgrimage80]

When I examine grimoirelab/grimoirelab:latest docker logs, I see the error I shared in the link. How can I solve this problem?

in docker it goes from up to exit.

945f836d644e grimoirelab/grimoirelab:latest "/bin/sh -c ${DEPLOY…" 20 hours ago Exited (1) About an hour ago dockercompose_mordred_1

https://pastebin.com/tpYmxBFz

[jjmerchante]

Thank you @pilgrimage80 for providing the logs.

I think the configuration files you provided don't correspond to the errors. There are two errors in the configuration files.

• This first error indicates that your setup.cfg file is wrong:

  Error while consuming configuration: Wrong type for section param: gitlab:issue studies <class 'str'> should be <class 'list'>

  Your setup.cfg in the [gitlab:issue] section has something like

  studies = [enrich_onion:gitlab-issue] # (optional)

  or

  studies = enrich_onion:gitlab-issue

  In this case you should remove # (optional) or create a list:

  studies = [enrich_onion:gitlab-issue] 

• The second error is related to your projects.json file:

  json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 8 column 13 (char 251)

  It looks like you are using single quotes (') in your projects.json and must be substituted with double quotes (")

After fixing those errors try to restart the containers:

docker compose down
docker compose up -d

[pilgrimage80]

Now when I check the docker logs mordered I get the following error.

2024-03-22 09:34:08,651 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc1602f0a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F 2024-03-22 09:34:12,679 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3eb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F 2024-03-22 09:34:20,691 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3460>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F 2024-03-22 09:34:36,717 - GrimoireLab - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc15fe3e80>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /api/v4/projects/%2F 2024-03-22 09:34:36,748 - GrimoireLab - grimoire_elk.elk - ERROR - Error feeding raw from gitlab (https://gitlabtest//): HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn conn.connect() File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen return self.urlopen( File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen return self.urlopen( File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 827, in urlopen return self.urlopen( [Previous line repeated 2 more times] File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 799, in urlopen retries = retries.increment( File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/grimoire_elk/elk.py", line 205, in feed_backend ocean_backend.feed(params) File "/usr/local/lib/python3.8/site-packages/grimoire_elk/raw/elastic.py", line 234, in feed self.feed_items(items) File "/usr/local/lib/python3.8/site-packages/grimoire_elk/raw/elastic.py", line 250, in feed_items for item in items: File "/usr/local/lib/python3.8/site-packages/perceval/backend.py", line 314, in fetch self.client = self._init_client() File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 249, in _init_client return GitLabClient(self.owner, self.repository, self.api_token, File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 517, in init self._init_rate_limit() File "/usr/local/lib/python3.8/site-packages/perceval/backends/core/gitlab.py", line 736, in _init_rate_limit response = super().fetch(url) File "/usr/local/lib/python3.8/site-packages/perceval/client.py", line 143, in fetch response = self._fetch_from_remote(url, payload, headers, method, stream, auth) File "/usr/local/lib/python3.8/site-packages/perceval/client.py", line 174, in _fetch_from_remote response = self.session.get(url, params=payload, headers=headers, stream=stream, File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gitlabtest', port=443): Max retries exceeded with url: /api/v4/projects/%2F (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efc168e63d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) 2024-03-22 09:34:36,798 - GrimoireLab - grimoire_elk.elk - INFO - [gitlab] Done collection for https://gitlabtest// 2024-03-22 09:34:36,799 - GrimoireLab - sirmordred.task_collection - INFO - [gitlab:issues] collection finished for / 2024-03-22 09:34:36,800 - GrimoireLab - sirmordred.task_manager - ERROR - [gitlab:issues] Exception in Task Manager list index out of range Traceback (most recent call last):

[pilgrimage80]

It says no result found under gitlab issues on the dashboard.

My indices and aliases

GET _cat/indices yellow open .grimoirelab-sigils _xV71ns4SaSCH5wC2ISLCQ 5 1 22 0 68.6kb 68.6kb yellow open gitlab_issues_demo_enriched iTpgphdFTwu8oATTf7aORA 5 1 0 0 1.2kb 1.2kb yellow open gitlab_merges_demo_raw w1PRfqinT7qk_oEBYs4BNA 5 1 0 0 1.2kb 1.2kb green open kibana_sample_data_ecommerce d4rcO8D4Sg-c-lF_jmj4_w 1 0 4675 0 4.8mb 4.8mb green open .kibana_1 wegrmJ1jS7OAiFM2QtnVqA 1 0 149 5 281.8kb 281.8kb green open .kibana_2 1ex7EVW5R0yjQ1A081IF2g 1 0 149 3 244.1kb 244.1kb yellow open gitlab_issues_demo_raw z2tcWKefS6Wp6v6zJpabRQ 5 1 0 0 1.2kb 1.2kb yellow open gitlab_merges_demo_enriched CYZsCUoIQAW33DtlExBj-A 5 1 0 0 1.2kb 1.2kb green open .kibana_4 oveHB8eXTeC2kxoJsXXVWQ 1 0 149 3 243.9kb 243.9kb green open .kibana_3 Uz-xo88kRAGStxGM9Q1mwQ 1 0 149 3 249.3kb 249.3kb yellow open api D0PN4l8BS4aKHPGrFetxLA 5 1 1 0 5.4kb 5.4kb

GET _cat/aliases

gitlab-raw gitlab_issues_demo_raw - - - gitlab_merge_requests-raw gitlab_merges_demo_raw - - - .kibana .kibana_4 - - -

[jjmerchante]

If your projects.json contains https://gitlabtest/t/xx.git, and your setup.cfg has enterprise-url = https://gitlabtest/, it won't collect anything because that domain doesn't exist.

Consider using a repository URL from which you want to collect data, such as https://gitlab.com/gitlab-org/grafana-dashboards in your projects.json. If it's the official GitLab instance, please remove the enterprise-url line from the setup.cfg, if it is an enterprise instance, replace gitlabtest with the actual domain.