search

groupe-sii / sonar-web-frontend-plugin

Sonar plugin to add metrics for web technologies (JS, CSS, HTML, SASS, AngularJS...)
Apache License 2.0
115 stars 26 forks source link

Impossible to start SonarQube with rules having description longer than 32766 characters (UTF-8) #19

Closed cpoisnel closed 8 years ago

cpoisnel commented 8 years ago

It's impossible to start Sonarqube 5.1+ (tested with 5.2) with this plugin. The rules can't be indexed by Elastic Search because descriptions are too long.

[130]: index [rules], type [rule], id [jshint:W090], message [IllegalArgumentException[Document contains at least one immense term in field="htmlDesc" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[60, 115, 116, 121, 108, 101, 62, 35, 99, 117, 115, 116, 111, 109, 45, 114, 117, 108, 101, 45, 100, 101, 115, 99, 114, 105, 112, 116, 105, 111]...', original message: bytes can be at most 32766 in length; got 32980]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 32980]; ]
    at org.sonar.server.search.IndexQueue.enqueue(IndexQueue.java:94) ~[sonar-server-5.2.jar:na]
    at org.sonar.db.DbSession.commit(DbSession.java:61) ~[sonar-db-5.2.jar:na]
    at org.sonar.server.rule.RegisterRules.start(RegisterRules.java:106) ~[sonar-server-5.2.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
    at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89) ~[picocontainer-2.15.jar:na]
    at org.sonar.core.platform.ComponentContainer$1.start(ComponentContainer.java:259) ~[sonar-core-5.2.jar:na]
    at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.behaviors.Stored.start(Stored.java:110) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009) ~[picocontainer-2.15.jar:na]
    at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767) ~[picocontainer-2.15.jar:na]
    at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:99) ~[sonar-core-5.2.jar:na]
    at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:84) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.platformlevel.PlatformLevelStartup.access$001(PlatformLevelStartup.java:45) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:82) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:78) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:197) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.Platform.doStart(Platform.java:114) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.Platform.doStart(Platform.java:99) ~[sonar-server-5.2.jar:na]
    at org.sonar.server.platform.PlatformServletContextListener.contextInitialized(PlatformServletContextListener.java:43) ~[sonar-server-5.2.jar:na]
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4720) [tomcat-embed-core-8.0.18.jar:8.0.18]
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154) [tomcat-embed-core-8.0.18.jar:8.0.18]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [tomcat-embed-core-8.0.18.jar:8.0.18]
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409) [tomcat-embed-core-8.0.18.jar:8.0.18]
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1399) [tomcat-embed-core-8.0.18.jar:8.0.18]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

Workaround

aurelien-baudet commented 8 years ago

The plugin is not compatible since Sonar 5.0 for now as there are many internal changes in plugin development and bugs. Many new APIs are marked as beta and may change in next versions. So we decided to wait before supporting Sonar 5+. I will investigate the source of the issue and see if Sonar has a simple way to bypass elasticsearch indexing on description

aurelien-baudet commented 8 years ago

I have made some researches. There is no way to easily exclude some fields. The problem has been introduced since 5.1: https://jira.sonarsource.com/browse/SONAR-6632. Currently, there is no fix for this issue in Sonar. It seems to be planned for version 5.3.

I have reduced the size of the HTML description. I also had to fix some calls to Sonar to make it work. I have not tested completely the plugin on Sonar 5.2 but it seems to work.

@cpoisnel : Please let me know if it works for you

cpoisnel commented 8 years ago

Thank you Aurélien, I will test it :+1:

mslama-work commented 8 years ago

Please what table/rules shall I clean from DB? I tested this plugin but this issue persists even if I deleted this plugin from my testing Sonar instance. Just FYI: I deleted record with: delete from rules where plugin_rule_key = 'W083', Sonar starts ok after this.

aurelien-baudet commented 8 years ago

Which version of Sonar are you using ?

mslama-work commented 8 years ago

Sonar 5.4

aurelien-baudet commented 8 years ago

I have tested sonar-5.x branch on Sonar 5.4 and Sonar starts correctly.

Have you generated the plugin using branch sonar-5.x ? The main branch works for sonar < 5. Due to big changes into internal Sonarqube APIs, the plugin that works for sonar 4.5.x can't work in Sonar 5+. So we had to make different branches.

Can you try again generating the plugin with branch sonar-5.x and start with it ?

mslama-work commented 8 years ago

I used plugin from release 2.0.0 zip file. I will test plugin branch sonar-5.x with Sonar 5.4 and let you know. I must also say Sonara start correctly for the first time. But next start runs indexing on data added by plugin after first start. So second start failed due to Sonaer issue SONAR-6632 mentioned above. This issue should be fixed in Sonar 5.6 - there is already Sonar 5.6 RC1 available but as you say there is many more changes in plugin required for Sonar 5.x.

We also have clash with Sonar Web plugin. It uses the same source file filter and project build fails because it complains there are 2 plugins with the same source filter. So it was reason I had to remove web-frontend plugin. By default both plugins are enabled when installed and we cannot go through all projects and update their configuration. Is there any way how to enable web-frontend plugin only for some projects explicitly? ie. we would use either web or web-frontend plugin.

aurelien-baudet commented 8 years ago

It is no more possible to have two plugins that are able to handle same language or same kind of files... I had the same issue when using both our plugin and Sonar javascript plugin.

I tried many things, I also ask Sonar how I could do this and their response is clear : there is no more way and they won't do anything to help us...

So I tried differently by including their plugin into ours. It partially works and you can have at the same time Sonar Way rules and our rules. But it is a really long work to make it fully working (including test results, code duplication, code structure...). So for now, we prefer to wait for the Sonar 5.6 release that will be a LTS (will be available soon). We will investiguate for a real working solution when available.

So for now, it is not possible to have both plugins at the same time on the same Sonarqube server isntance.

gfrancesca commented 7 years ago

@aurelien-baudet hi! Have you some news about usage of both javascript sonar plugin and your sonar-web-frontend? (for sonar 5.6.. ) Thanks Francesca