search

GoogleCloudPlatform / fluent-plugin-detect-exceptions

A fluentd plugin that scans line-oriented log streams and combines exceptions stacks into a single log entry.
Apache License 2.0
192 stars 69 forks source link

Not able to detect sql based stacktrace/exceptions #61

Closed vkperumal closed 4 years ago

vkperumal commented 4 years ago

Hi We are using fluentd for shipping logs from EKS to splunk and using fluent-plugin-detect-exceptions for detecting exception and combine them to single log.

Normal exceptions are getting detected and shipped as single event but sql based java exception gets divided and sent to splunk as individual lines.

Normal Exception (working): javax.ws.rs.NotFoundException: Unable to find Opportunity at com.tactile.kingslayer.metadata.TableManager.retrieve(TableManager.java:109) at com.tactile.kingslayer.objects.ObjectInstanceManager.getObjectRecord(ObjectInstanceManager.java:324) at com.tactile.kingslayer.api.InstanceApi.getObjectInstance(InstanceApi.java:86) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:377) at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) at java.base/java.lang.Thread.run(Thread.java:834)

SQL java exception (NOT working):

2019-10-08 17:36:44 ERROR c.t.k.e.ExceptionMappers ID=36e69553-7c85-4711-9a2e-ef004953c565 Uncaught exception, returning as 400 DATA_ERROR org.jooq.exception.DataAccessException: SQL [insert into "data_divisions_4" (timezones, divisions, parentids, name, locales, shortnames, "id", "createdbys", "lastmodifiedbys", isdeleteds, createddates, lastmodifieddates, systemlastmodifieddate__s) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, cast(current_timestamp as timestamp with time zone), cast(current_timestamp as timestamp with time zone), cast(current_timestamp as timestamp with time zone)) returning divisions, createdbys, createddates, lastmodifiedbys, name, lastmodifieddates, id, systemlastmodifieddates, isdeleteds]; ERROR: duplicate key value violates unique constraint "UNIQUE_INDEX_data_divisions_4_shortnames" Detail: Key (shortname__s)=(IBM) already exists. at org.jooq_3.11.9.POSTGRES_10.debug(Unknown Source) at org.jooq.impl.Tools.translate(Tools.java:2429) at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:832) at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:364) at org.jooq.impl.InsertImpl.fetchOne(InsertImpl.java:1061) at com.tactile.kingslayer.objects.ObjectInstanceManager.createObject(ObjectInstanceManager.java:219) at com.tactile.kingslayer.objects.ObjectInstanceManager.createObject(ObjectInstanceManager.java:97) at com.tactile.kingslayer.api.DivisionApi.lambda$createDivision$1(DivisionApi.java:99) at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:521) at org.jooq.impl.Tools$10$1.block(Tools.java:4422) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) at org.jooq.impl.Tools$10.get(Tools.java:4419) at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:564) at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:491) at com.tactile.kingslayer.api.DivisionApi.createDivision(DivisionApi.java:95) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:377) at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "UNIQUE_INDEX_data_divisions_4_shortnames" Detail: Key (shortname__s)=(IBM) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at org.jooq.tools.jdbc.DefaultPreparedStatement.executeQuery(DefaultPreparedStatement.java:94) at org.jooq.impl.AbstractDMLQuery.execute(AbstractDMLQuery.java:738) at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:350) ... 37 common frames omitted

This fails due to the line which comes in between (Detail: Key (shortname__s)=(IBM) already exists.)

Can you please suggest if there as workaround to detect this as well and ship as single log

jkohen commented 4 years ago

Thanks for the report. This plugin is intended to be used with Stackdriver. We don't know enough about ELK and Splunk to help.