Correct use of variables?

[magicofit]

I tested now the gitlab-integration with the following values:

NEW_RELIC_API_KEY -> I used a NR-user-api key GLAB_TOKEN -> i used a gitlab group access token with read_api rights and role "developer". GLAB_STANDALONE=False -> My expectation was, that wenn this variable is set to "False", then there is no internal scheduler used and only a single execution will happen...but this was not the case. GLAB_EXPORT_PATHS="groupwithrights" -> i added here a git-group name, where the GLAB_TOKEN is inside GLAB_EXPORT_PROJECTS_REGEX=".*", GLAB_LOW_DATA_MODE=True correct paths to gitlab&NR ect. With Version 1.0.2

So i executed my plan, to start with minimal data first and extend when we need more data and understand the impact.

...but i get no data in New Relic. But a very fast response "Exporter finished!!! Next job run in 61 minutes" Maybe more log-infos (or documentation) would be helpful to guide, what could be wrong here. :)

[dpacheconr]

You need NR license key for ingest If scheduler ran with GLAB_STANDALONE=false than that is bug, although if you running this as schedule in gitlab, you don't even need to set this variable as it's false by default , I will look at correct it Due to the bug you not seeing any messaging, once I correct the bug you should be able to see it

[dpacheconr]

try release 1.03 it should be fixed, don't forget the ingest license not user license

[magicofit]

Ok i tested 1.0.3&licensekey and i made some steps forward....here some feedback for the variables:

GLAB_PROJECT_OWNERSHIP=True If i use a gitlab-group-token and not i private one, then there will be no ownership in any case...and the default setting (True) prevent to get any data. A solution could be, that "GLAB_PROJECT_OWNERSHIP" is a optional parameter and will only be used, if existing.

GLAB_PROJECT_VISIBILITY="private" Expectation: Was a little bit unclear. My guess was, that "private" includes "internal" and "public" ...and "public" would include only "public". But it turned out, that "private" includes only "private" so in default setting i got no data, because 80% of our projects are "public" und 20% are "internal". Maybe a comma separated list would be a solution like: GLAB_PROJECT_VISIBILITY="internal,public" with default setting GLAB_PROJECT_VISIBILITY="private" so there is no breaking change. (or as a optional parameter with no filter as default)

GLAB_EXPORT_LOGS=False Expectation: I don't see any activities in the console that logs would be sent to NR if i set this variable to "False". Current Status: i got entries like "Log events sent for project: 123"

Gathering pipeline data for project 699 this may take while...
2023-03-15 11:32:28,584 INFO [project_logger] [get_resources.py:46] [trace_id=0 span_id=0 resource.service.name=] - Project: 699 - mylittle/project
Log events sent for project: 699 - mylittle/project
...
Failed to export logs, error code: StatusCode.PERMISSION_DENIED

GLAB_STANDALONE=False With 1.0.3 i got still a internal scheduler if i setted to "False"....but without this variable at all it works as expected.

what else ...and i got additionally some other "401: 401 Unauthorized" during the export-try (maybe connected to gitlab-group-token usage?).

Still no data in new relic ...but i know i try some things differently.

I hope, that feedback could help a bit. (I am not a python-developer but i can do some QA for the blackbox :) )

[dpacheconr]

Thanks for testing it. Do you have any public projects that you are not owner off, that you would like to monitor? Unfortunately, the Gitlab API does not support multiline visibility types in same request. So we can either run the exporter with different configurations or code around the limitation. Exporting logs will export your jobs stdout. Log events are also used to send other resources data. Most issues I have seen were all due to use group tokens. I wonder if this exporter should only support private tokens.... I will look at the GLAB_STANDALONE=False, but with new version I can no longer replicate it

[magicofit]

Nope, in our company are very few owners. Most developers are "Developer", "Maintainer" and sometimes "Reporter". So our main combinations are "Public"&"No Rights" and "Public"&"Developer".

For GLAB_PROJECT_VISIBILITY&GLAB_PROJECT_OWNERSHIP from my perspective it would be good, if they are handled like "filtering only used if existing". But i understand the gitlab.com-usecase and there the current default settings make sense.

But i have a 1.0.3 solution for me for this part: fist run via external scheduler: GLAB_PROJECT_VISIBILITY="public",GLAB_PROJECT_OWNERSHIP=False, without GLAB_STANDALONE second run via external scheduler: GLAB_PROJECT_VISIBILITY="internal",GLAB_PROJECT_OWNERSHIP=False, without GLAB_STANDALONE

But i dont understand the use of GLAB_EXPORT_LOGS ...i will prevent to send stdout job-logs to NR because, i cant be sure, that a developer wrote something like "echo password" somewhere in the project-pipeline. This was the reason, why i want to set it to "False". But as i understood, that job-logs are important for this gitlab-implementation?

And for the group-token / private-token issue: For me it would make sense if private&user-tokens are not the first choice on both sides (NR and gitlab), because if someone leave the company the tokens are invalid. So i would tend to suggest user Independent tokens not only on NR-side.

[dpacheconr]

Hi

GLAB_EXPORT_LOGS For logs in context, it will require the logs to be sent to NR, however, the integration will still work without it, you will still be able to see your pipelines/jobs as traces and it's metrics.

For GLAB_PROJECT_VISIBILITY&GLAB_PROJECT_OWNERSHIP, I could make some changes to make it configurable for multiple visibilities in same run( code around API limitation), however, what happens when users would also want to change ownership each time they change visibility, i.e. public with ownership false, then on second run it with internal and ownership true. To cater for this, both visibility and ownership will have to be configurable together per each run.

Makes sense to keep support for both group and personal token. As long as users are aware that their configuration needs to reflect on what type of token they are using.

Apart from the good to have feature of different configurations per run of the exporter, which currently is only possible by running the exporter multiple times each with it's own configuration.

Is there any other outstanding issue you experiencing with this integration?

Regards

Diogo Pacheco Senior Solutions Architect EMEA Field Engineering Team www.newrelic.com http://www.newrelic.com

On Mon, 20 Mar 2023 at 09:46, magicofit @.***> wrote:

Nope, in our company are very few owners. Most developers are "Developer", "Maintainer" and sometimes "Reporter". So our main combinations are "Public"&"No Rights" and "Public"&"Developer".

For GLAB_PROJECT_VISIBILITY&GLAB_PROJECT_OWNERSHIP from my perspective it would be good, if they are handled like "filtering only used if existing". But i understand the gitlab.com-usecase and there the current default settings make sense.

But i have a 1.0.3 solution for me for this part: fist run via external scheduler: GLAB_PROJECT_VISIBILITY="public",GLAB_PROJECT_OWNERSHIP=False, without GLAB_STANDALONE second run via external scheduler: GLAB_PROJECT_VISIBILITY="internal",GLAB_PROJECT_OWNERSHIP=False, without GLAB_STANDALONE

But i dont understand the use of GLAB_EXPORT_LOGS ...i will prevent to send stdout job-logs to NR because, i cant be sure, that a developer wrote something like "echo password" somewhere in the project-pipeline. This was the reason, why i want to set it to "False". But as i understood, that job-logs are important for this gitlab-implementation?

And for the group-token / private-token issue: For me it would make sense if private&user-tokens are not the first choice on both sides (NR and gitlab), because if someone leave the company the tokens are invalid. So i would tend to suggest user Independent tokens not only on NR-side.

— Reply to this email directly, view it on GitHub https://github.com/newrelic-experimental/gitlab/issues/12#issuecomment-1475912732, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYBQSMNL6L3FSIX2AN2DG43W5ARPPANCNFSM6AAAAAAV2GWE3A . You are receiving this because you commented.Message ID: @.***>

[magicofit]

Yes, even if i do all correct with all workarounds, i see no data in new relic....but i see the script try something.

combination: gitlab-group-token (with developer-role) with GLAB_EXPORT_PATHS="thisgroup"

log results (sometimes shortened with ...) in attachment

[dpacheconr]

Hi

If we disregard the errors collecting data from gitlab, which are expected depending on the token you use, we can see some events of attempts to export data to New Relic, but with some errors on export ( not on collecting from gitlab )

Are you using a New Relic License Ingest Key? If you are, can you check there are no spaces before or after or any missing characters? The only time I saw those errors were when the license key was incomplete.

I have just replicated this on my test environment, by removing one character from the license key I had set up as an environment variable.

Happy to set up a call to review your setup if it helps.

Regards

Diogo Pacheco Senior Solutions Architect EMEA Field Engineering Team www.newrelic.com http://www.newrelic.com

On Tue, 21 Mar 2023 at 10:59, magicofit @.***> wrote:

Yes, even if i do all correct with all workarounds, i see no data in new relic....but i see the script try something.

combination: gitlab-group-token (with developer-role) with GLAB_EXPORT_PATHS="thisgroup"

log results https://github.com/newrelic-experimental/gitlab/files/11027477/log.txt (sometimes shortened with ...) in attachment [image: dashboard] https://user-images.githubusercontent.com/36575071/226586495-27ba26ca-5801-46bb-9906-bcddcf38fd0d.PNG

— Reply to this email directly, view it on GitHub https://github.com/newrelic-experimental/gitlab/issues/12#issuecomment-1477633874, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYBQSMM3R245VSKFYUAY2ETW5GCYBANCNFSM6AAAAAAV2GWE3A . You are receiving this because you commented.Message ID: @.***>

[magicofit]

YES! Thanks!! good hint! I used a CI-variable before ...-e NEW_RELIC_API_KEY="${NEW_RELIC_INGEST_TOKEN}" (but i am pretty sure, that there was no additional space). but now i tried -e NEW_RELIC_API_KEY="cleartexttoken" and it worked! I have still no clue, why CI-variable didnt work, but now i have my first data. happy

[dpacheconr]

I am glad its working for you. I added extra logging on 1.0.4, if you ran it, it should output to the logs what repositories is failing to obtain information closing this ticket, any issues let us know