search

hygieia / hygieia-codequality-sonar-collector

Hygieia Collector for Sonar versions 5.x to 8.x
https://github.com/Hygieia/Hygieia/blob/gh-pages/pages/hygieia/collectors/build/sonar.md
Apache License 2.0
4 stars 52 forks source link

Sonar Collector: Unable to pull in projects beyond the first page (500 records) #12

Closed irwindouglas closed 4 years ago

irwindouglas commented 5 years ago

Looks like Sonar has set a max page size of 500 on api/projects as well now. Experimenting out our Sonar system with 1333 projects, nothing is returned from the collector beyond the first page. Can we add paging support to have the collector loop through all pages increment the pageIndex until all records are collected?

Related to: https://github.com/Hygieia/Hygieia/issues/933

irwindouglas commented 5 years ago

After pulling the code I think I see the issue. There is paging already on this. It calculated the total records and divides by the PAGE_SIZE and rounds up to the correct number of pages needed to pull the records. When it pulls the record it adds &p=pageNumber but from what I can tell it doesn't send the pagesize. The max page size is 500, however, when I ran this query without setting the page size it looks like the default is 100. so it only returns 100 records at a time but the for loop is assuming 500 records at a time so only 1/5 of the records are returned. My suggestion is to either pass in the pageSize in the url as well or reduce the PAGE_SIZE to 100 to use the default.

irwindouglas commented 5 years ago

well, just found the code that is setting the pagesize as well when appending URL_RESOURCES. Now I am not sure why we are only seeing records from the first 500 projects...

irwindouglas commented 5 years ago

Reduced our total sonar projects (pruned branches and older projects) to 500 and our project starting showing up. Still tracking down the issue but definitely was not showing in the auto-complete before in the UI widget

irwindouglas commented 5 years ago

In collector_items filtering on com.capitalone.dashboard.model.SonarProject only 491 records exist but if I query Sonar directly there are 543 records. I would expect these to match so there must be something causing the records to not fully process. Looking through the error logs only show a few warning on a cookie it doesn't like but no errors.

Sbrenthughes commented 5 years ago

@fidelcoria-aa is this something you can check into and possibly provide a fix if needed?

alzafacon commented 5 years ago

Currently working on a PR for sonarqube unrelated to this. I'll give this issue a stab after I've finished that PR, if it has not been fixed yet.

alzafacon commented 5 years ago

@irwindouglas FYI the sonarqube collector code has moved https://github.com/Hygieia/hygieia-codequality-sonar-collector I checked issue Hygieia/Hygieia#933 and that is about the DefaultSonarClient which is for some versions of sonarQube. IDK what version of sonarqube you are using.. really I'm not sure what is causing your problem. I would say try using the latest version of sonarqube and see if it works. I am using sonarqube 6.7.2 without any problems

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.