search

hygieia / hygieia-scm-gitlab-collector

Hygieia Collector for Gitlab SCM
https://github.com/capitalone/Hygieia/blob/gh-pages/pages/hygieia/collectors/scm/gitlab.md
Apache License 2.0
4 stars 13 forks source link

hygieia-scm-gitlab-collector is not pulling old commits/issues #6

Open arghyasinha19 opened 5 years ago

arghyasinha19 commented 5 years ago

gitlab collector is not pulling old commits. Below is my app.properties file. I have downloaded the latest collector from github.

dbname=hygieia dbhost=localhost dbport=27017 gitlab.cron=0 0/1 gitlab.host=gitlab..net gitlab.protocol=https gitlab.port=443 gitlab.selfSignedCertificate=true gitlab.apiToken=xxccffghhha gitlab.apiVersion=4 gitlab.key=xxccffghhha gitlab.useProjectId = true gitlab.commitThresholdDays=15

Please find the log below.

2019-08-15 14:55:01,397 [taskScheduler-1] INFO c.c.d.c.GitlabGitCollectorTask - {password=, personalAccessToken=, branch=UAT, userID=xxxEiqghfxxxxn, url= https://gitlab.xxxxxx.net/206}::UAT:: get commits 2019-08-15 14:55:01,398 [taskScheduler-1] INFO c.c.d.gitlab.GitlabUrlUtility - projectIdValue: 206 2019-08-15 14:55:01,398 [taskScheduler-1] INFO c.c.d.gitlab.GitlabUrlUtility - ---> Gitlab commits URI: https://gitlab.xxx.net:443/api/v4/projects/206/rep ository/commits/?ref_name=UAT&since=2019-08-15T18:44:01.453Z&per_page=100 2019-08-15 14:55:01,446 [taskScheduler-1] INFO c.c.d.gitlab.DefaultGitlabGitCli ent - page 1: 200 2019-08-15 14:55:01,448 [taskScheduler-1] INFO c.c.d.c.GitlabGitCollectorTask - {password=, personalAccessToken=, branch=UAT, userID=xxxxxxxNufkMvRn, url= https://gitlab.xxxxxx.net/206}::UAT get issues 2019-08-15 14:55:01,476 [taskScheduler-1] INFO c.c.d.gitlab.GitlabUrlUtility - ---> Gitlab issues URI: https://gitlab.xxxxxx.net:443/api/v4/projects/206/issu es/?scope=all&created_after=2019-08-15T18:44:01.453Z&per_page=100&order_by=updat ed_at&sort=desc 2019-08-15 14:55:01,512 [taskScheduler-1] INFO c.c.d.c.GitlabGitCollectorTask - {password=, personalAccessToken=, branch=UAT, userID=xxxxxxxufkMvRn, url= https://gitlab.xxxxxx.net/206}::UAT::get pulls 2019-08-15 14:55:01,520 [taskScheduler-1] INFO c.c.d.gitlab.GitlabUrlUtility - ---> Gitlab merge requests URI: https://gitlab.xxxxxx.net:443/api/v4/projects/ 206/merge_requests/?state=all&created_after=2019-08-15T18:44:01.453Z&per_page=10 0&scope=all&order_by=updated_at&sort=desc 2019-08-15 14:55:01,578 [taskScheduler-1] INFO c.c.d.collector.CollectorTask - Repo Count 1 0s 2019-08-15 14:55:01,579 [taskScheduler-1] INFO c.c.d.collector.CollectorTask - New Commits 0 0s 2019-08-15 14:55:01,579 [taskScheduler-1] INFO c.c.d.collector.CollectorTask - New Issues 0 0s 2019-08-15 14:55:01,579 [taskScheduler-1] INFO c.c.d.collector.CollectorTask - New Pulls 0 0s 2019-08-15 14:55:01,580 [taskScheduler-1] INFO c.c.d.collector.CollectorTask - Finished 0s

arghyasinha19 commented 5 years ago

Please note I am behind corporate proxy but all proxy related handle has been added.

rvema commented 5 years ago

@zhenweiding Any thoughts on the above issue?

zhenweiding commented 5 years ago

2019-08-15 14:55:01,398 [taskScheduler-1] INFO c.c.d.gitlab.GitlabUrlUtility - ---> Gitlab commits URI: https://gitLab.xxxx.net:443/api/v4/projects/206/rep ository/commits/?ref_name=UAT&since=2019-08-15T18:44:01.453Z&per_page=100

@arghyasinha19 I notice the since timestamp is later than the log's timestamp, could you give me more information about your server's time zone settings?

arghyasinha19 commented 5 years ago
TimeStamp

Please note, it was working properly with Hygieia 2.0. After I recently upgraded to Hygieia 3.0 and started getting the issue.

arghyasinha19 commented 5 years ago

The screenshot shows the timezone of Hygieia server. I checked now and it seems Gitlab server is using a different time zone (GMT).

zhenweiding commented 5 years ago

@arghyasinha19 Please check this code(L436), if not the first time to run, the collector only fetch the data from 10 minutes ago of last updated date.

https://github.com/Hygieia/hygieia-scm-gitlab-collector/blob/master/src/main/java/com/capitalone/dashboard/gitlab/GitlabUrlUtility.java#L423

arghyasinha19 commented 5 years ago

So it means if I am running for the first time it will only show the commits done 10 mins back...other past commits will not be displayed.

arghyasinha19 commented 5 years ago

@zhenweiding also could you pls update the url to gitLab.xxxx.net in your first comment...thanks in advance.

zhenweiding commented 5 years ago

So it means if I am running for the first time it will only show the commits done 10 mins back...other past commits will not be displayed.

No！If running for the first time it will fetch the data from 14 days ago.

Please check this code:

https://github.com/Hygieia/hygieia-scm-gitlab-collector/blob/master/src/main/java/com/capitalone/dashboard/gitlab/GitlabUrlUtility.java#L427

and this:

https://github.com/Hygieia/hygieia-scm-gitlab-collector/blob/master/src/main/java/com/capitalone/dashboard/collector/GitlabGitCollectorTask.java#L123 https://github.com/Hygieia/hygieia-scm-gitlab-collector/blob/master/src/main/java/com/capitalone/dashboard/collector/GitlabGitCollectorTask.java#L186

wufei523 commented 3 years ago

the issue is the sample properties file's gitlab.commitThresholdDays=15 doesnt match whats's in code: https://github.com/Hygieia/hygieia-scm-gitlab-collector/blob/master/src/main/java/com/capitalone/dashboard/collector/GitlabSettings.java#L93

if you change gitlab.commitThresholdDays=15 to gitlab.firstRunHistoryDays=15, it will work