[mbox] Threads analysis stopped on https://mail.gnome.org/archives/gnome-list

[valeriocos]

An error is thrown when analysing https://mail.gnome.org/archives/gnome-list using the metric newthreads.NewThreadsHistoricMetricProvider from 01/01/1998 to 31/12/2018. More details at http://crossminer.bitergia.net:5601/#/project/configure/gnomelist7.

ERROR [MetricListExecutor (gnomelist7, 19990109)] (20:20:39): Exception thrown during metric provider execution (historic.newsgroups.newthreads).
com.mongodb.MongoException: WiredTigerIndex::insert: key too large to index, failing  1418 { : "https://mail.gnome.org/archives/gnome-list/:Mbox        <3696B197.351ED9E8@ihug.co.nz>  Robert Napier wrote:
 > You're missing the critical logfile line. Go ch..." }
        at com.mongodb.CommandResult.getException(CommandResult.java:100)
        at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
        at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
        at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:249)
        at com.mongodb.DBApiLayer$MyCollection.createIndex(DBApiLayer.java:344)
        at com.mongodb.DBCollection.createIndex(DBCollection.java:467)
        at com.mongodb.DBCollection.ensureIndex(DBCollection.java:543)
        at com.googlecode.pongo.runtime.PongoCollection.createIndex(PongoCollection.java:27)
        at org.eclipse.scava.metricprovider.trans.newsgroups.threads.model.ThreadDataCollection.<init>(ThreadDataCollection.java:11)
        at org.eclipse.scava.metricprovider.trans.newsgroups.threads.model.NewsgroupsThreadsTransMetric.setDb(NewsgroupsThreadsTransMetric.java:40)
        at org.eclipse.scava.metricprovider.trans.newsgroups.threads.model.NewsgroupsThreadsTransMetric.<init>(NewsgroupsThreadsTransMetric.java:13)
        at org.eclipse.scava.metricprovider.trans.newsgroups.threads.ThreadsTransMetricProvider.adapt(ThreadsTransMetricProvider.java:103)
        at org.eclipse.scava.metricprovider.historic.newsgroups.newthreads.NewThreadsHistoricMetricProvider.measure(NewThreadsHistoricMetricProvider.java:74)
        at org.eclipse.scava.platform.MetricHistoryManager.store(MetricHistoryManager.java:36)
        at org.eclipse.scava.platform.osgi.analysis.MetricListExecutor.run(MetricListExecutor.java:106)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
WARN  [ProjectExecutor (w1:gnomelist7:onlythreads)] (20:20:39): Project in error state. Resuming execution.

[creat89]

Commit https://github.com/crossminer/scava/commit/953284bb4c7661bde065989e0496192ef604f38e should fix the issue. The problem was a very large subject that was breaking the limit in MongoDB for a key

[valeriocos]

The error has changed and now the task is marked with an error (http://crossminer.bitergia.net:5601/#/worker) @tdegueul any idea about it?

WARNING: Exception or error caught in server resource
Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:540)
        at org.restlet.resource.ServerResource.get(ServerResource.java:743)
        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:618)
        at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:679)
        at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:357)
        at org.restlet.resource.ServerResource.handle(ServerResource.java:1044)
        at org.restlet.resource.Finder.handle(Finder.java:236)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Router.doHandle(Router.java:422)
        at org.restlet.routing.Router.handle(Router.java:639)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
        at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
        at org.restlet.Application.handle(Application.java:385)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Router.doHandle(Router.java:422)
        at org.restlet.routing.Router.handle(Router.java:639)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.routing.Router.doHandle(Router.java:422)
        at org.restlet.routing.Router.handle(Router.java:639)
        at org.restlet.routing.Filter.doHandle(Filter.java:150)
        at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
        at org.restlet.routing.Filter.handle(Filter.java:197)
        at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
        at org.restlet.Component.handle(Component.java:408)
        at org.restlet.Server.handle(Server.java:507)
        at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
        at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
        at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:273)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoException: Executor error: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
        at com.mongodb.MongoException.parse(MongoException.java:82)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:293)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:274)
        at com.mongodb.DBCursor._check(DBCursor.java:368)
        at com.mongodb.DBCursor._hasNext(DBCursor.java:459)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:484)
        at org.eclipse.scava.platform.visualisation.Chart.createDatatable(Chart.java:82)
        at org.eclipse.scava.platform.visualisation.MetricVisualisation.visualise(MetricVisualisation.java:80)
        at org.eclipse.scava.platform.client.api.MetricVisualisationResource.doRepresent(MetricVisualisationResource.java:75)
        at org.eclipse.scava.platform.client.api.AbstractApiResource.represent(AbstractApiResource.java:55)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
        ... 45 more

[creat89]

I guess you issue has been solved, so I'll close it. If you face again an issue, don't hesitate in opening it again.

[creat89]

The issue indicated in https://github.com/crossminer/scava/issues/442#issuecomment-561708307 is related to the creation of visualization files. And it might be related to https://github.com/crossminer/scava/issues/447