SonarOpenCommunity / sonar-cxx

SonarQube C++ Community plugin (cxx plugin): This plugin adds C++ support to SonarQube with the focus on integration of existing C++ tools.
GNU Lesser General Public License v3.0
988 stars 364 forks source link

Coverage sensor: GC overhead limit exceeded #1340

Closed tovishwendra closed 6 years ago

tovishwendra commented 6 years ago

I am trying to import bullseye coverage report using sonar cxx community plugin. In sonar qube i am not able to see which of the conditions are covered/uncovered, even though it shows the percent of the condition coverage. For line coverage it shows which of the functions are covered/uncovered by highlighting the covered ones with green color and uncovered ones with red color, but same difference its not showing for condition coverage.

Below are the versions of softwares used: SonarQube: 6.1 SonarCxxcommunityplugin: 0.9.7 Bullseye format: 8.9.94

guwirth commented 6 years ago

Hi @tovishwendra please try with 0.9.8. There are many improvements for coverage. https://github.com/SonarOpenCommunity/sonar-cxx/releases/tag/cxx-0.9.8

tovishwendra commented 6 years ago

Hi guwirth, As per your suggestion we have upgraded sonar cxx plugin to 0.9.8 and after uploading coverage report of a component in sonarqube i am able to see if a condition is covered or not in sonar qube. But when i am trying to upload the coverage report of the whole product i am getting an exception while sonar analysis as "GC overhead limit exceeded" with SONAR_SCANNER_OPTS set to 4096 as per the command set SONAR_SCANNER_OPTS=-Xmx12288m. After this i increased the value of VM to 12 GB using command set SONAR_SCANNER_OPTS=-Xmx12288m and sonar analysis went on for 10 hrs and after that it started showing Java heap space error as below "caused By: java.lang.outofmemoryerror: java heap space"

guwirth commented 6 years ago

Hi @tovishwendra,

never heard from such problems? What's different in your project?

Regards,

tovishwendra commented 6 years ago

Hi guwirth,

After moving back to sonar cxx plugin 0.9.7 i am able to upload the product level report to sonar server but the whole process of sonar scanner analysis and uploading of report took approx 7 hrs. There is no change in anything except that plugin has been downgraded from 0.9.8 to 0.9.7 in sonar server.

With plugin version 0.9.8 i was sometimes seeing GC overhead limit exceeded error, sometime java heap space error and sometimes sonar scanner was still continuing even after 24 hrs.

Please find below answers to your query. 1) How big is your code in LOC? As calculate by utility LocMetrics LOC has come out to be 10657413 minus 2.6 million since there are some source files common in difference components. So approx 7 I have also attached the screenshot of it. screenshotloc

2) How big is your coverage file (MB), is it one big file or more than one file? We are using 3 coverage files for single sonar scanner analysis and upload of UT , IT and overall coverage. UT file size: 17MB IT file size: 17MB Overall file size: 17MB 3) Is the exception on scanner side or on server side? I am not sure where the exception side or server side but in the stack trace i can see the below line which i think is from the plugin side.

org.sonar.cxx.sensors.squid.CxxSquidSensor.execute(CxxSquidSensor.java:185)

4) Could you provide more information to the exception (stack trace)?

22866, 22867, 22868, 22869, 22870, 22871, 22872, 22873, 22874, 22875, 22876, 22877, 22878, 22879, 22880, 22881, 22882, 22883, 22884, 22885, 22886, 22887, 22888, 22889, 22890, 22891, 22892, 22893, 22894, 22895, 22896, 22897, 22898, 22899, 22900, 22901, 22902, 22903, 22904, 22905, 22906, 22907, 22908, 22909, 22910]' 16:21:13.465 DEBUG: linesOfComments: '[]' 16:24:56.749 DEBUG: global settings for: 'F:\BLR.ovxpl.11.30.i\winx64\ccover\hpsw-sc\OvXpl\BBN\codesets\include\EucTW.h' 16:24:56.780 DEBUG: Parse 'F:\BLR.ovxpl.11.30.i\winx64\ccover\hpsw-sc\OvXpl\BBN\codesets\include\EucTW.h' as C++ file Exception in thread "JOURNAL_FLUSHER" Exception in thread "LOG_FLUSHER" Exception in thread "CLEANUP_MANAGER" 20:12:28.360 WARN: [CHECKPOINT_WRITER] WARNING Transaction neither committed nor rolled back Transaction_100000002 depth=1 status= Exception in thread "CHECKPOINT_WRITER" java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.(ArrayList.java:152) at com.persistit.CleanupManager.poll(CleanupManager.java:166) at com.persistit.CleanupManager.runTask(CleanupManager.java:88) at com.persistit.IOTaskRunnable.run(IOTaskRunnable.java:144) at java.lang.Thread.run(Thread.java:745) java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.iterator(ArrayList.java:834) at java.util.Collections$SynchronizedCollection.iterator(Collections.java:2031) at com.persistit.Persistit.pollAlertMonitors(Persistit.java:2285) at com.persistit.Persistit$LogFlusher.run(Persistit.java:192) java.lang.OutOfMemoryError: Java heap space at java.util.regex.Matcher.(Matcher.java:226) at java.util.regex.Pattern.matcher(Pattern.java:1093) at java.util.Formatter.parse(Formatter.java:2547) at java.util.Formatter.format(Formatter.java:2501) at java.util.Formatter.format(Formatter.java:2455) at java.lang.String.format(String.java:2940) at com.persistit.logging.PersistitLogMessage$Enabled.logMessage(PersistitLogMessage.java:248) at com.persistit.logging.PersistitLogMessage$Enabled.log(PersistitLogMessage.java:266) at com.persistit.logging.PersistitLogMessage$LogDispatchHandler.log(PersistitLogMessage.java:152) at com.persistit.JournalManager$JournalFlusher.runTask(JournalManager.java:2402) at com.persistit.IOTaskRunnable.run(IOTaskRunnable.java:144) at java.lang.Thread.run(Thread.java:745) java.lang.OutOfMemoryError: Java heap space 20:12:29.171 INFO: ------------------------------------------------------------------------ 20:12:29.171 INFO: EXECUTION FAILURE 20:12:29.171 INFO: ------------------------------------------------------------------------ 20:12:29.171 INFO: Total time: 9:15:21.733s 20:12:37.611 INFO: Final Memory: 230M/11557M 20:12:37.611 INFO: ------------------------------------------------------------------------ 20:12:37.611 ERROR: Error during SonarQube Scanner execution org.sonar.squidbridge.api.AnalysisException: Unable to parse file: F:\BLR.ovxpl.11.30.i\winx64\ccover\hpsw-sc\OvXpl\BBN\codesets\include\EucTW.h at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:105) at org.sonar.cxx.sensors.squid.CxxSquidSensor.execute(CxxSquidSensor.java:185) at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:53) at org.sonar.scanner.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:57) at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:49) at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:78) at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:175) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:262) at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:257) at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:247) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:47) at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:86) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:118) at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:117) at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60) at com.sun.proxy.$Proxy0.execute(Unknown Source) at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233) at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151) at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:123) at org.sonarsource.scanner.cli.Main.execute(Main.java:77) at org.sonarsource.scanner.cli.Main.main(Main.java:61) Caused by: java.lang.OutOfMemoryError: Java heap space at org.sonar.sslr.internal.matchers.ParseNode.(ParseNode.java:39) at org.sonar.sslr.internal.vm.Machine.createNode(Machine.java:256) at org.sonar.sslr.internal.vm.Instruction$RetInstruction.execute(Instruction.java:305) at org.sonar.sslr.internal.vm.Machine.execute(Machine.java:162) at org.sonar.sslr.internal.vm.Machine.execute(Machine.java:105) at org.sonar.sslr.internal.vm.Machine.parse(Machine.java:61) at com.sonar.sslr.impl.Parser.parse(Parser.java:84) at com.sonar.sslr.impl.Parser.parse(Parser.java:69) at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:94) at org.sonar.cxx.sensors.squid.CxxSquidSensor.execute(CxxSquidSensor.java:185) at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:53) at org.sonar.scanner.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:57) at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:49) at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:78) at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:175) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:262) at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:257) at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:247) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:47) at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:86) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143) at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128) at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:118) at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:117) at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 20:12:37.611 DEBUG: Execution getVersion 20:12:37.751 DEBUG: Execution stop

I would also like to know how can we reduce the sonar scanner analysis to take less time. Now its taking around 7 hrs.

guwirth commented 6 years ago

@tovishwendra that's really difficult to say what the route cause is. With 0.9.8 the squid sensor is doing several things (https://github.com/SonarOpenCommunity/sonar-cxx/blob/cxx-0.9.8/cxx-sensors/src/main/java/org/sonar/cxx/sensors/squid/CxxSquidSensor.java#L150):

To speed up coverage there is a coverage cache to avoid reading the coverage file for each source code file again. With 6.1 there are still these UT / IT / overall coverage values.... So can be that this coverage cache is the reason or the depth of scanning (too many includes).

I would also like to know how can we reduce the sonar scanner analysis to take less time.

Another try is to use the 0.9.9 snapshot (but working only with SQ 6.7).

tovishwendra commented 6 years ago

Hi Guwirth, Few points that i have here.

1) Sonar server to which the upload is failing with plugin version 0.9.8 is of version 6.3. If we want to upload to sonar server 6.3 which coverage report(UT,IT, Overall) and which property should i give in sonar-project.properties file to upload to sonar server 6.3 because (UT, IT and Overall) will have different coverage measurent as per their test cases.

2) Finally for the project release i will have to upload to sonar server 6.1. Management has not yet decided to move to upgraded sonar server. Sonar server 6.1 support (UT, IT and Overall) reports to be uploaded to sonar. As per your suggestion above you have asked to do the below things. 2.1) Your Point: First of all you should not provide includes to system headers. Mostly they are not needed. My Query: Can you please elaborate on this point or provide a link for it. 2.2) Your Point: Other possibility is to split project into smaller sub-projects My Query: Are you talking about multi module sonar project? 2.3) You have mentioned that coverage cache can be the reason of the failure. My Query: How can we avoid coverage cache?

It will be great help if you can provide insights on above points.

guwirth commented 6 years ago

@tovishwendra I’m sorry see less possibilities. 2.1: You can try to provide less paths sonar.cxx.includeDirectories 2.2: idea was more component A is SQ project A, component B is ... 2.3: with current implementation is not possible to avoid caching