wildcard default search not working

[ryneal]

I have a self-hosted Gitlab instance, currently running 13.9.1.

When the search parameter is set to an empty string (or left default), it is resulting in the following issue;

DEBU[2021-05-24T11:09:23Z] listing all projects from wildcard            wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind= wildcard-owner-name= wildcard-search=
WARN[2021-05-24T11:09:29Z] taskqtask="getProjectsFromWildcardTask" failed (will retry=1 in dur=30s): unable to list projects with search pattern '' from the GitLab API : GET https://<gitlab_url>/api/v4/projects: 500 {message: 500 Internal Server Error}
INFO[2021-05-24T11:09:53Z] scheduling metrics pull                       environments-count=0 refs-count=0

After analysis of the codebase, I have uncovered that the search value is sent if other wildcard parameters are not identified https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/blob/main/pkg/gitlab/projects.go#L80

It may be worth considering an additional case where if the string is not set, not to include it in the request?

[mvisonneau]

:wave: hey @ryneal, thank you for raising this issue. I, unfortunately, do not have a self-hosted v13.9.1 to try it out. However, I successfully tested the following config over a v13.11.0 & v13.12.1 with the following parameters:

# with a group
wildcards:
  - owner: { name: foo, kind: group, include_subgroups: true }

# with an empty wildcard
wildcards:
  - {}

In both cases, it was using the token of an admin user with api scope / v0.4.9 of the exporter

Could you please share more details regarding your configuration and the type of token you are using? :thinking:

[awx-fuyuanchu]

Same issue here

exporter version: mvisonneau/gitlab-ci-pipelines-exporter:v0.5.1

Gitlab version: 14.1.2

full config:

log:
  format: json
gitlab:
  url: http://xxx-xxx.xxx.xxx/
  enable_health_check: true
  enable_tls_verify: true
wildcards: 
  - {}

related logs:

{"level":"info","msg":"starting 'metrics' garbage collection","time":"2021-09-18T01:38:05Z"}
{"level":"info","msg":"ending 'metrics' garbage collection","time":"2021-09-18T01:38:05Z"}
{"level":"info","msg":"scheduling refs from projects pull","projects-count":0,"time":"2021-09-18T01:38:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:38:05Z"}
{"level":"info","msg":"scheduling environments from projects pull","projects-count":0,"time":"2021-09-18T01:38:05Z"}
{"level":"info","msg":"scheduling projects from wildcards pull","time":"2021-09-18T01:38:05Z","wildcards-count":1}
{"level":"info","msg":"starting 'refs' garbage collection","time":"2021-09-18T01:38:05Z"}
{"level":"info","msg":"ending 'refs' garbage collection","time":"2021-09-18T01:38:05Z"}
{"level":"warning","msg":"taskqtask=\"PullProjectsFromWildcard\" handler failed after retry=1: unable to list projects with search pattern '' from the GitLab API : GET https://xxx-xxx.xxx.xxx/api/v4/projects: 500 {message: 500 Internal Server Error}","time":"2021-09-18T01:38:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:38:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:39:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:39:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:40:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:40:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:41:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:41:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:42:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:42:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:43:05Z"}
{"level":"info","msg":"starting 'metrics' garbage collection","time":"2021-09-18T01:43:05Z"}
{"level":"info","msg":"ending 'metrics' garbage collection","time":"2021-09-18T01:43:05Z"}
{"level":"info","msg":"scheduling refs from projects pull","projects-count":0,"time":"2021-09-18T01:43:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:43:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:44:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:44:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:45:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:45:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:46:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:46:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:47:05Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:47:35Z"}
{"environments-count":0,"level":"info","msg":"scheduling metrics pull","refs-count":0,"time":"2021-09-18T01:48:05Z"}
{"level":"info","msg":"starting 'metrics' garbage collection","time":"2021-09-18T01:48:05Z"}
{"level":"info","msg":"ending 'metrics' garbage collection","time":"2021-09-18T01:48:05Z"}
{"level":"info","msg":"scheduling refs from projects pull","projects-count":0,"time":"2021-09-18T01:48:05Z"}

[dmpe]

Jop, can also confirm with version 0.5.2. Same as above

time="2021-10-25T14:15:49Z" level=error msg="Key: 'Config.Projects' Error:Field validation for 'Projects' failed on the 'at-least-1-project-or-wildcard' tag\nKey: 'Config.Wildcards' Error:Field validation for 'Wildcards' failed on the 'at-least-1-project-or-wildcard' tag"

EDIT: Solved in case somebody is using helm chart, then wildcards must be inside of config: key

[JanSchuermannPH]

Same issue than @FuyuanChu . exporter version: mvisonneau/gitlab-ci-pipelines-exporter:v0.5.2 Gitlab version 14.4.2-ee The problem occurs when no per_page parameter is set, you can try it in a browser. Anything you can do about it? Otherwise this tool is pretty useless for self-hosted instances...

[waderobson]

Found my way here too. Self hosted 14.5.3 Turns out I just needed to wait for it to try again.

[rudnypc]

Same problem here.