dpacheconr
commented
1 year ago Thank you @XC- for your feedback.
In summary following changes are recommend
- make the project filtering configurable for ownership and visibility as per https://python-gitlab.readthedocs.io/en/stable/gl_objects/projects.html
- make path part of the name filtering
- improve efficiency
We were testing this after seeing the blog post, and while the integration to a single pipeline worked decently, the metrics exporter was a no-go. It is still a bit unclear to me whether they are intended to be used together or as alternatives.
The first problem was that the script did not fetch any projects. This was due to the project listing having
owned=True. When disabled, I was able to fetch the projects properly.The problem is that the new-relic-metrics-exporter.py does a huge amount of requests to the Gitlab REST API and the longer the history, the more requests there will be (in my test case of ~15-20 repositories, this meant ~14 000 requests). For example, it fetches all deployments ever created (in our case there were ~17k objects in the response, meaning ~850 queries) and only after that filters which of those belong to the specified time window storing only the ids. After the filtering, the matching objects are fetched again. While this time there will be less requests, it is still inefficient when the objects could be stored and used from memory. This kind of inefficiency happens throughout the script, e.g. the project information is fetched in 11 locations in the code, meaning the same information might be requested from the server hundreds of times, or even more.
The group and project filtering is also not very optimal, since it does not provide a way to use the path as a filter and relies on the names intended for human use. That mean that if there is a same group name in two different paths, both are matches and could even have matching project names (while not as likely as the group name "conflict", it is made worse by the regex usage since those are prone to human errors and can match more places than intended). This makes the usage more difficult. Instead, there should be option to match the path e.g.
foo/bar/foobar, wherefooandbarare groups, andfoobara project.So while an interesting option and possibility, in the current form I cannot recommend the
new-relic-metrics-exporterat least to be used unless these inefficiencies are fixed somehow.