search

metatron-app / metatron-discovery

Powerful & Easy way for big data discovery
https://metatron.app
Apache License 2.0
441 stars 112 forks source link

error on ingesting large oracle table #2485

Open ctamietto opened 5 years ago

ctamietto commented 5 years ago

i'm trying to ingest a large oracle database . 1) when Select Data I choose Table 2) on ingestion settings I choose Ingest Once 3) at this point i try to ingest all the data table ( about 120000 rows ) and i get the error

the full stack trace follow :

Caused by: org.apache.derby.client.am.SqlException: Statement too complex. Try rewriting the query to remove complexity. Eliminating many duplicate expressions or breaking up the query and storing interim results in a temporary table can often help resolve this error. at org.apache.derby.client.am.ClientStatement.completeSqlca(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.parsePrepareError(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.parsePRPSQLSTTreply(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.StatementReply.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatement.readPrepareDescribeOutput_(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientStatement.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.readPrepareDescribeInputOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.flowPrepareDescribeInputOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.prepare(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientConnection.prepareStatementX(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:288) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:288) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.skife.jdbi.v2.DefaultStatementBuilder.create(DefaultStatementBuilder.java:52) ~[jdbi-2.63.1.jar:2.63.1] at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1304) ~[jdbi-2.63.1.jar:2.63.1] at org.skife.jdbi.v2.Query.iterator(Query.java:240) ~[jdbi-2.63.1.jar:2.63.1] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator.getTimelineForIntervalsWithHandle(IndexerSQLMetadataStorageCoordinator.java:255) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator.access$000(IndexerSQLMetadataStorageCoordinator.java:80) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator$1.withHandle(IndexerSQLMetadataStorageCoordinator.java:141) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator$1.withHandle(IndexerSQLMetadataStorageCoordinator.java:137) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1] ... 63 more Caused by: org.apache.derby.client.am.SqlException: Java class file format limit(s) exceeded: method:e2 codelength (103011 > 65535) in generated class org.apache.derby.exe.acc2db804dx016cx7c33x2d26x000033111e6020. at org.apache.derby.client.am.ClientStatement.completeSqlca(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.parsePrepareError(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.parsePRPSQLSTTreply(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatementReply.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.StatementReply.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.net.NetStatement.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientStatement.readPrepareDescribeOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.readPrepareDescribeInputOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.flowPrepareDescribeInputOutput(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientPreparedStatement.prepare(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientConnection.prepareStatementX(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown Source) ~[derbyclient-10.11.1.1.jar:?] at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:288) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:288) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.skife.jdbi.v2.DefaultStatementBuilder.create(DefaultStatementBuilder.java:52) ~[jdbi-2.63.1.jar:2.63.1] at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1304) ~[jdbi-2.63.1.jar:2.63.1] at org.skife.jdbi.v2.Query.iterator(Query.java:240) ~[jdbi-2.63.1.jar:2.63.1] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator.getTimelineForIntervalsWithHandle(IndexerSQLMetadataStorageCoordinator.java:255) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator.access$000(IndexerSQLMetadataStorageCoordinator.java:80) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator$1.withHandle(IndexerSQLMetadataStorageCoordinator.java:141) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at io.druid.metadata.IndexerSQLMetadataStorageCoordinator$1.withHandle(IndexerSQLMetadataStorageCoordinator.java:137) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]

kyungtaak commented 5 years ago

@ctamietto Can you tell me the segment granularity information you set? image

You should set the "Segment Granularity" option depending on the range and density of data in the timestamp column. From the current error log, it seems that many intervals have been created by setting the "Segment Granularity" option. If possible, try to make the granularity unit wider. (e.g hour-> month)