CSV export of large results can lead to unresponsive web interface process

[kroepke]

Users report that after several large CSV exports the web interface can become unresponsive requiring a restart.

Investigate if repeated canceling of exports can lead to accumulated garbage in the thread.

[arjen9999]

We are experiencing the same issue. Even without using any proxy for graylog web, after some csv exports the web service becomes unresponsible. Only a restart of the graylog-webservice makes things work again.

Really hope this can be fixed in 1.3.0 since it makes our environment very unstable at the moment.

[arjen9999]

Not sure if helpful, but many of the below messages show up in the application.log around the time that the graylog-webservice becomes unresponsive.

2015-11-30T18:42:11.611Z - [ERROR] - from org.graylog2.restclient.lib.ApiClient in pool-222-thread-1 API call timed out java.util.concurrent.TimeoutException: null at com.ning.http.client.providers.netty.future.NettyResponseFuture.get(NettyResponseFuture.java:159) ~[com.ning.async-http-client-1.9.31.jar:na] at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.executeOnAll(ApiClientImpl.java:608) ~[org.graylog2.graylog2-rest-client--1.2.2-1.2.2.jar:na] at controllers.api.MetricsController$PollingJob.run(MetricsController.java:117) [graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_66] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_66] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_66] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

[zexxon]

Can you please provide some context around the CSV export you are doing? i.e. are you trying to export days of data or just a few hundred log messages? Have you tried using the REST API directly or only used the web interface?

CSV as a raw data format doesn't work well for large volumes of data, especially tools like Excel which have hard limits for analyzing data based on number of raw cells etc. If the use case is for further analysis, we may provide a batch mechanism for breaking up the number of messages to avoid hanging the web interface or look at providing JDBC integration instead.

[arjen9999]

Users try to export a couple of thousand rows of data for offline analysis indeed. Did not try rest api directly, but just restarting the graylog-webservice is enough to restabilize. Graylog server doesnt need a restart.

Even just a configurable limit for amount of rows that can be exported would be a great start. That can stabilize at least a production environment from heavy users :)

[zexxon]

We'll continue to look at the issue then. In the mean time, you may want to setup another web server instance purely for managing CSV exports. Not ideal but it's another option.

We are planning on adding query limit support in the 2.0 release, which may help by constraining the volume of data returned.

[arjen9999]

Thank you! I hope the query limit can be set at a different than export limit. Export limit would be i guess a much lower value. Let me know if any more information can be provided to assist.

[JakeDEvans]

Version 2.0.2, I was able to export 1,454,441 messages in 2,159 ms, searched in 2 indices. containing 6 headers (not message) to a csv totaling 270 MB

I'd consider this resolved!

[edmundoa]

Good to hear that @JakeDEvans!

I'm closing the issue, as it is quite outdated (there's no web interface process on Graylog 2.0, other than a web browser), and it seems that it works better than before on this regard. Please consider open a new issue in the graylog2-server repository if this is an issue after upgrading to 2.0.x.