Admin stats on NON enterprise account?

[luciano-buono]

Hi, I enabled the flag GITHUB_EXPORTER_COLLECTOR_ADMIN with the idea of collecting the metrics related to the admin stats (available to ORG aswell). However, the exporter seems to make the api call as:

{"collector":"admin","err":"GET https://api.github.com/enterprise/stats/all: 404 Not Found []","level":"error","msg":"Failed to fetch admin stats","ts":"2023-08-31T14:46:37.877816288Z"}

Is it possible to have the ADMIN stats in a not enterprise account?

[tboerger]

Currently it's not possible, I didn't even realized that these starts could be available for orgs as well. Sounds like a great feature of the stats are not already part of the org metrics.

[luciano-buono]

Noted! Thanks for the confirmation

Sorry to use this issue for another topic, but I'm having a problem with the WORKFLOW collector.

I have enabled the flag GITHUB_EXPORTER_COLLECTOR_WORKFLOWS: true and defined multiple repos (which many have a workflow history), however I don't see any metrics about that.

The pod has been running for 17h so it already should have gathered info about it. (Also there are not ERROR logs about anything

Also in a strange note, in the metrics I see this:

# HELP github_request_failures_total Total number of failed requests to the api per collector.
# TYPE github_request_failures_total counter
github_request_failures_total{collector="action"} 0
github_request_failures_total{collector="billing"} 0
github_request_failures_total{collector="org"} 0 

And I havent enabled the actions collector, but I enabled the workflow collector.. Here are my env vars defined in the helm chart:

extraEnvVariables:
  GITHUB_EXPORTER_ORGS: XX
  GITHUB_EXPORTER_REPOS: XX

  GITHUB_EXPORTER_COLLECTOR_BILLING: true

  # This admin stats are only for enterprise accounts
  GITHUB_EXPORTER_COLLECTOR_ADMIN: false
  # Not needed
  GITHUB_EXPORTER_COLLECTOR_REPOS: false

  # Information about orgs and repos runners. If they are busy or idle
  GITHUB_EXPORTER_COLLECTOR_RUNNERS: false

  # Check timeouts
  GITHUB_EXPORTER_COLLECTOR_WORKFLOWS: true
  # Needed to avoid context deadline exceeded errors in collector
  GITHUB_EXPORTER_REQUEST_TIMEOUT: 5m
  GITHUB_EXPORTER_PER_PAGE: 100
  GITHUB_EXPORTER_WEB_TIMEOUT: 3m

  GITHUB_EXPORTER_WEB_PPROF: true

[tboerger]

On the start of the exporter you should see a log line if the workflow exporter is enabled, after that you should be able to see recent workflow metrics of enabled repos.

[luciano-buono]

This are all the logs from k logs deployments/promhippie-github-exporter:

orangepi@orangepi5:~/Documents/kubernetes/promhippie-github-exporter$ k logs deployments/promhippie-github-exporter 
{"date":"20230820","go":"go1.21.0","level":"info","msg":"Launching GitHub Exporter","revision":"42a4ade","ts":"2023-08-31T20:12:08.672852603Z","version":"2.4.0"}
{"addr":"0.0.0.0:9504","level":"info","msg":"Starting metrics server","ts":"2023-08-31T20:12:08.673292722Z"}
{"address":"[::]:9504","level":"info","msg":"Listening on","ts":"2023-08-31T20:12:08.673944296Z"}
{"address":"[::]:9504","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2023-08-31T20:12:08.673984254Z"}

EDIT: I have enabled the debug flag GITHUB_EXPORTER_LOG_LEVEL: debug and see that the workflow collector is enabled. I'll wait to see if I get any info with that

[kirillbilchenko]

@luciano-buono I facing the same issue, not clear how to make this working

[luciano-buono]

@kirillbilchenko Which issue do u mean? the GITHUB_EXPORTER_COLLECTOR_ADMIN is not possible currently.

For the workflows metrics, it's sorta working for me, I needed to add repos with current workflows runs. The collector looks like it request information from the last workflow run, not a list with previous runs.

[luciano-buono]

@kirillbilchenko How many repos do u have?

What values did you put in:

  GITHUB_EXPORTER_REQUEST_TIMEOUT: 10m
  GITHUB_EXPORTER_PER_PAGE: 100
  GITHUB_EXPORTER_WEB_TIMEOUT: 10m

I'm getting a lot of Rate limiting by Github and can't figure out the right values

[kirillbilchenko]

@luciano-buono sorry seems I commented in the wrong thread, so in total I have 500+ repos, yes, after adding the correct parameters, it's working fine now, the problem was for me confusing name in parameters, about the enterprise metrics, I tried and also got the same errors as you.

[luciano-buono]

@kirillbilchenko would you mind sharing your parameters?

I also have +500 repos and can't find a way to balance between not having stale metrics on Prometheus and not rate limiting the request to Github

[tboerger]

Guys it would be great to keep issues separated. If you got issues with the workflow collector take a look at https://github.com/promhippie/github_exporter/issues/218, this issue should be focused on admin stats for non-enterprise access.

[kirillbilchenko]

GITHUB_EXPORTER_WEB_TIMEOUT

so in the end I enabled only for most important repos, because was not able to make it working in the right way with all repos.

[tboerger]

In the end I don't see a way how to solve this issue. I can't prevent the rate limits by GitHub.

[kirillbilchenko]

In the end I don't see a way how to solve this issue. I can't prevent the rate limits by GitHub.

just an idea, maybe in this case, it make sense to add the webhook listener, and configure the webhooks on org level? and just listen what github will give to us? it make sense to have some kind of filtering what to process what to drop(having some kind of filtering)

[tboerger]

I'm currently thinking about how to reimplement the whole workflow metrics feature.

Currently I'm tending to integrate a webhook listener which got to be configured on repos or orgs as a target, store the state with boltdb or optionally some other database and use the database content to generate the metrics offered by this exporter.

It would be great to gather some feedback by people using this feature at this larger scale as this would turn the exporter into some stateful service.

[tboerger]

@luciano-buono could you explain a little more details about this feature request? Which API endpoints are you refering to? Currently we are using https://github.com/google/go-github/blob/v56.0.0/github/admin_stats.go#L156 to fetch the statistics for a GitHub ES which is documented at https://docs.github.com/en/enterprise-server@3.10/rest/enterprise-admin/admin-stats?apiVersion=2022-11-28#get-all-statistics.