search

change-metrics / monocle

Monocle helps teams and individual to better organize daily duties and to detect anomalies in the way changes are produced and reviewed.
https://changemetrics.io
GNU Affero General Public License v3.0
377 stars 58 forks source link

What needs to be done to get metrics on private repos in an organization? #1005

Closed dannydi12 closed 1 year ago

dannydi12 commented 1 year ago

I've tried everything from personal access tokens, to a 3rd party github app. Am I missing something?

Every time, the logs don't output any errors. Nothing seems awry, site loads. Just no data.

Can someone outline what I need to do to get metrics on a private repo in my github organization?

Thank you!

morucci commented 1 year ago

It should works with a personal access token with the "repo" scope. Also if it is a personal Github account then repositories must be listed in "github_repositories" setting.

Also better if you could share an anonymized logs and the config file.

dannydi12 commented 1 year ago

Sorry for the delay. Yes! Here are my config files:

image image

So far, I've created a token with no scopes, all scopes, and just the repo scope and it still didn't work.

So sorry for being a noob, but how would I generate those anonymized logs for you?

morucci commented 1 year ago

Hi,

I've tried with a private repo from an organization I own and Monocle was able to crawl PRs from it, so I don't know what is missing in your configuration because that looks as expected. The personal token must be with the "repo" scope.

Here is the log from my crawler process:

2022-12-22 10:04:35 INFO    Macroscope.Worker:168: Processing {"index":"monocle-org","crawler":"private-repo-org-crawler","stream":"Changes","entity":{"contents":"monocle-org/demo-repo","tag":"Project"},"age":"2015-01-01T00:00:00Z"}
2022-12-22 10:04:37 INFO    Lentille.GraphQL:220: Fetched from current page {"index":"monocle-org","crawler":"private-repo-org-crawler","stream":"Changes","count":1,"total":1,"pageInfo":{"endCursor":"Y3-...-Ql7","hasNextPage":false,"totalCount":1},"ratelimit":{"remaining":4740,"resetAt":"2022-12-22T10:51:45Z","used":260}}
2022-12-22 10:04:37 INFO    Macroscope.Worker:173: Posting documents {"index":"monocle-org","crawler":"private-repo-org-crawler","stream":"Changes","count":1}

Could you do a "docker-compose logs crawler" and look for similar logs ?

Also please note that the default query use "from:now-3weeks" by default, so it might be possible the PRs on your private repo are older than 3 weeks. Perhaps try with from:now-500weeks.

dannydi12 commented 1 year ago

Thank you for the help!

Yes, the token has the repo scope and I generated it from this URL: https://github.com/settings/tokens

The logs:

monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Main:86: Starting monitoring service {"port":9001}
monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Main:113: Starting to fetch streams
monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Main:179: Group starting {"group":"https://api.github.com/graphql-[REDACTED] for test"}
monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Main:304: Starting crawler {"index":"test","crawler":"test"}
monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Main:165: Starting crawlers {"crawlers":["https://api.github.com/graphql-[REDACTED] for test"]}
monocle-crawler-1  | 2022-12-22 17:02:01 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"TaskDatas","offset":0}
monocle-crawler-1  | 2022-12-22 17:02:01 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":0,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:01 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":1,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:03 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":2,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:05 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":3,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:09 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":4,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:17 WARNING Monocle.Effects:520: network error {"index":"test","crawler":"test","stream":"TaskDatas","count":5,"limit":7,"loc":"api:8080/api/2/crawler/get_commit_info","failed":"ConnectionFailure Network.Socket.connect: <socket: 40>: does not exist (Connection refused)"}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:168: Processing {"index":"test","crawler":"test","stream":"TaskDatas","entity":{"contents":"SocialPod/social-pod","tag":"TaskDataEntity"},"age":"2019-01-01T00:00:00Z"}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:164: Crawling entities completed {"index":"test","crawler":"test","stream":"TaskDatas","entity":{"contents":"SocialPod/social-pod","tag":"TaskDataEntity"},"age":"2022-12-22T17:02:01Z"}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"Changes","offset":0}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:168: Processing {"index":"test","crawler":"test","stream":"Changes","entity":{"contents":"SocialPod/social-pod","tag":"Project"},"age":"2019-01-01T00:00:00Z"}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Lentille.GraphQL:220: Fetched from current page {"index":"test","crawler":"test","stream":"TaskDatas","count":0,"total":0,"pageInfo":{"endCursor":null,"hasNextPage":false,"totalCount":0},"ratelimit":{"remaining":4868,"resetAt":"2022-12-22T17:54:12Z","used":132}}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:186: Continuing on next entity {"index":"test","crawler":"test","stream":"TaskDatas"}
monocle-crawler-1  | 2022-12-22 17:02:33 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"TaskDatas","offset":0}
monocle-crawler-1  | 2022-12-22 17:02:34 INFO    Lentille.GraphQL:220: Fetched from current page {"index":"test","crawler":"test","stream":"Changes","count":0,"total":0,"pageInfo":{"endCursor":null,"hasNextPage":false,"totalCount":0},"ratelimit":{"remaining":4867,"resetAt":"2022-12-22T17:54:12Z","used":133}}
monocle-crawler-1  | 2022-12-22 17:02:34 INFO    Macroscope.Worker:186: Continuing on next entity {"index":"test","crawler":"test","stream":"Changes"}
monocle-crawler-1  | 2022-12-22 17:02:34 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"Changes","offset":0}
monocle-crawler-1  | 2022-12-22 17:02:34 INFO    Macroscope.Worker:164: Crawling entities completed {"index":"test","crawler":"test","stream":"Changes","entity":{"contents":"SocialPod/social-pod","tag":"Project"},"age":"2022-12-22T17:02:34Z"}
monocle-crawler-1  | 2022-12-22 17:02:34 INFO    Macroscope.Main:182: Group end {"group":"https://api.github.com/graphql-[REDACTED] for test"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Main:179: Group starting {"group":"https://api.github.com/graphql-[REDACTED] for test"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Main:304: Starting crawler {"index":"test","crawler":"test"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Lentille.GraphQL:220: Fetched from current page {"index":"test","crawler":"test","stream":"TaskDatas","count":0,"total":0,"pageInfo":{"endCursor":null,"hasNextPage":false,"totalCount":0},"ratelimit":{"remaining":4861,"resetAt":"2022-12-22T17:54:12Z","used":139}}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:186: Continuing on next entity {"index":"test","crawler":"test","stream":"TaskDatas"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"TaskDatas","offset":0}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:164: Crawling entities completed {"index":"test","crawler":"test","stream":"TaskDatas","entity":{"contents":"SocialPod/social-pod","tag":"TaskDataEntity"},"age":"2022-12-22T17:12:37Z"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"Changes","offset":0}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"TaskDatas","offset":0}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:168: Processing {"index":"test","crawler":"test","stream":"Changes","entity":{"contents":"SocialPod/social-pod","tag":"Project"},"age":"2022-12-22T17:02:34Z"}
monocle-crawler-1  | 2022-12-22 17:12:37 INFO    Macroscope.Worker:168: Processing {"index":"test","crawler":"test","stream":"TaskDatas","entity":{"contents":"SocialPod/social-pod","tag":"TaskDataEntity"},"age":"2022-12-22T17:02:01Z"}
monocle-crawler-1  | 2022-12-22 17:12:38 INFO    Macroscope.Worker:186: Continuing on next entity {"index":"test","crawler":"test","stream":"Changes"}
monocle-crawler-1  | 2022-12-22 17:12:38 INFO    Macroscope.Worker:155: Looking for oldest entity {"index":"test","crawler":"test","stream":"Changes","offset":0}
monocle-crawler-1  | 2022-12-22 17:12:38 INFO    Macroscope.Worker:164: Crawling entities completed {"index":"test","crawler":"test","stream":"Changes","entity":{"contents":"SocialPod/social-pod","tag":"Project"},"age":"2022-12-22T17:12:38Z"}
monocle-crawler-1  | 2022-12-22 17:12:38 INFO    Macroscope.Main:182: Group end {"group":"https://api.github.com/graphql-[REDACTED] for test"}
monocle-crawler-1  | 2022-12-22 17:12:38 INFO    Lentille.GraphQL:220: Fetched from current page {"index":"test","crawler":"test","stream":"Changes","count":0,"total":0,"pageInfo":{"endCursor":null,"hasNextPage":false,"totalCount":0},"ratelimit":{"remaining":4860,"resetAt":"2022-12-22T17:54:12Z","used":140}}
morucci commented 1 year ago

According to your logs it seems this is working normally. However it seems your repo does not own pull requests updated after january 2019. I see count and total to 0.

dannydi12 commented 1 year ago

It seems to be working now. I think I might have had a typo for the repo name and accidentally selected another repo in the organization by some fluke. So sorry, thank you for your time!

morucci commented 1 year ago

No problem. Closing the issue.