search

TIBCOSoftware / snappydata

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
http://www.snappydata.io
Other
1.04k stars 201 forks source link

column table with 'KEY_COLUMN' can't delete existing old entry #1014

Open crabo opened 6 years ago

crabo commented 6 years ago

I'm using jdbc sql "insert into ys_send values(item_1..),.....(item_1000,.....)" batch insertion.

(Server=/10.2.6.26[1528] Thread=ThriftProcessor-12) The exception 'com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 The putAll operation failed to put 29 out of 1000 entries. ' was thrown while evaluating an expression.

-----------------------------------nested------------------------- java.sql.SQLException: (SQLState=38000 Severity=30000) (Server=10.2.6.26[1528] Thread=ThriftProcessor-12) The exception 'com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 The putAll operation failed to put 29 out of 1000 entries. ' was thrown while evaluating an expression. at io.snappydata.thrift.SnappyDataService$executePreparedUpdate_result$executePreparedUpdate_resultStandardScheme.read(SnappyDataService.java:13242) ~[snappydata-store-shared-1.6.1.jar:na] at io.snappydata.thrift.SnappyDataService$executePreparedUpdate_result$executePreparedUpdate_resultStandardScheme.read(SnappyDataService.java:13219) ~[snappydata-store-shared-1.6.1.jar:na] at io.snappydata.thrift.SnappyDataService$executePreparedUpdate_result.read(SnappyDataService.java:13158) ~[snappydata-store-shared-1.6.1.jar:na] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[libthrift-0.9.3.jar:0.9.3] at io.snappydata.thrift.SnappyDataService$Client.recv_executePreparedUpdate(SnappyDataService.java:403) ~[snappydata-store-shared-1.6.1.jar:na] at io.snappydata.thrift.SnappyDataService$Client.executePreparedUpdate(SnappyDataService.java:387) ~[snappydata-store-shared-1.6.1.jar:na] at io.snappydata.thrift.internal.ClientService.executePreparedUpdate(ClientService.java:1140) ~[snappydata-store-client-1.6.1.jar:na] at io.snappydata.thrift.internal.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:254) ~[snappydata-store-client-1.6.1.jar:na] at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384) ~[c3p0-0.9.5.2.jar:0.9.5.2] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task.execute(SnappyJdbcWriter.java:240) [snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task.doBulkInsert(SnappyJdbcWriter.java:413) [snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task.access$100(SnappyJdbcWriter.java:101) [snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task$2.call(SnappyJdbcWriter.java:437) ~[snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task$2.call(SnappyJdbcWriter.java:434) ~[snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil$Retry.call(RetryUtil.java:164) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:111) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30) [datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task.doJobWithRetry(SnappyJdbcWriter.java:434) [snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.writer.snappywriter.SnappyJdbcWriter$Task.startWrite(SnappyJdbcWriter.java:333) [snappywriter-jdbc-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

-----------------------------------nested------------------------- Caused by: java.rmi.ServerException: Server STACK: java.sql.SQLException(38000): The exception 'com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 The putAll operation failed to put 29 out of 1000 entries. ' was thrown while evaluating an expression. at com.pivotal.gemfirexd.internal.iapi.error.StandardException.newException(StandardException.java:473) at com.pivotal.gemfirexd.internal.engine.Misc.processGemFireException(Misc.java:1046) at com.pivotal.gemfirexd.internal.engine.Misc.processFunctionException(Misc.java:831) at com.pivotal.gemfirexd.internal.engine.Misc.processFunctionException(Misc.java:784) at com.pivotal.gemfirexd.internal.engine.Misc.processKnownGemFireException(Misc.java:1007) at com.pivotal.gemfirexd.internal.engine.Misc.processGemFireException(Misc.java:1018) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer.insertMultipleRows(GemFireContainer.java:3604) at com.pivotal.gemfirexd.internal.engine.sql.execute.GemFireInsertResultSet.handleMultipleInserts(GemFireInsertResultSet.java:306) at com.pivotal.gemfirexd.internal.engine.sql.execute.GemFireInsertResultSet.open(GemFireInsertResultSet.java:210) at com.pivotal.gemfirexd.internal.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:593) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:2175) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1984) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedCallableStatement.executeStatement(EmbedCallableStatement.java:133) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:393) at io.snappydata.thrift.server.SnappyDataServiceImpl.executePreparedUpdate(SnappyDataServiceImpl.java:2495) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate.getResult(SnappyDataService.java:1631) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate.getResult(SnappyDataService.java:1615) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at io.snappydata.thrift.server.SnappyDataServiceImpl$Processor.process(SnappyDataServiceImpl.java:201) at io.snappydata.thrift.server.SnappyThriftServerThreadPool$WorkerProcess.run(SnappyThriftServerThreadPool.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at io.snappydata.thrift.server.SnappyThriftServer$1.lambda$newThread$0(SnappyThriftServer.java:143) at java.lang.Thread.run(Thread.java:745)

-----------------------------------nested------------------------- Caused by: com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 The putAll operation failed to put 29 out of 1000 entries. at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend(PartitionedRegion.java:2498) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.postPutAll(LocalRegionDataView.java:406) at com.gemstone.gemfire.internal.cache.TXStateProxy.postPutAll(TXStateProxy.java:3843) at com.gemstone.gemfire.internal.cache.LocalRegion.basicPutAll(LocalRegion.java:11846) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer.insertMultipleRows(GemFireContainer.java:3539) ... 17 more

-----------------------------------nested------------------------- Caused by: com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 at com.pivotal.gemfirexd.internal.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:314) at com.pivotal.gemfirexd.internal.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:287) at com.pivotal.gemfirexd.internal.engine.distributed.utils.GemFireXDUtils.throwAssert(GemFireXDUtils.java:1921) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.handleNotDeleted(GfxdIndexManager.java:2749) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.deleteFromIndexes(GfxdIndexManager.java:2490) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.doDelete(GfxdIndexManager.java:2795) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.postEvent(GfxdIndexManager.java:1265) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:905) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:4421) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1985) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:8221) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:8193) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:87) at com.gemstone.gemfire.internal.cache.TXStateProxy$17.operateOnSharedDataView(TXStateProxy.java:4540) at com.gemstone.gemfire.internal.cache.TXStateProxy.performOp(TXStateProxy.java:3041) at com.gemstone.gemfire.internal.cache.TXStateProxy.destroyExistingEntry(TXStateProxy.java:1742) at com.gemstone.gemfire.internal.cache.BucketRegion.destroyAllEntries(BucketRegion.java:885) at com.gemstone.gemfire.internal.cache.BucketRegion.internalCreateAndInsertColumnBatch(BucketRegion.java:828) at com.gemstone.gemfire.internal.cache.BucketRegion.createAndInsertColumnBatch(BucketRegion.java:774) at com.gemstone.gemfire.internal.cache.partitioned.PutAllPRMessage.doLocalPutAll(PutAllPRMessage.java:689) at com.gemstone.gemfire.internal.cache.PartitionedRegion.tryToSendOnePutAllMessage(PartitionedRegion.java:3045) at com.gemstone.gemfire.internal.cache.PartitionedRegion.sendOrWaitForRespWithRetry(PartitionedRegion.java:2872) at com.gemstone.gemfire.internal.cache.PartitionedRegion.sendMsgByBucket(PartitionedRegion.java:2757) at com.gemstone.gemfire.internal.cache.PartitionedRegion.access$100(PartitionedRegion.java:239) at com.gemstone.gemfire.internal.cache.PartitionedRegion$8.execute(PartitionedRegion.java:2423) at com.gemstone.gnu.trove.THashMap.forEachEntry(THashMap.java:309) at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend(PartitionedRegion.java:2415) ... 21 more at io.snappydata.thrift.common.ThriftExceptionUtil.newSQLException(ThriftExceptionUtil.java:109) ~[snappydata-store-shared-1.6.1.jar:na] at io.snappydata.thrift.internal.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:262) ~[snappydata-store-client-1.6.1.jar:na] ... 13 common frames omitted

suranjan commented 6 years ago

Hi, Could you please tell me which kind of operations you were trying to do which you got this Exception?

On Thu, Apr 26, 2018 at 8:28 AM, crabo notifications@github.com wrote:

18/04/26 10:53:00.958 CST OplogCompactor SNAPPY-INTERNAL-DELTA for oplog oplog#717 INFO snappystore: OplogCompactor for SNAPPY-INTERNAL-DELTA compaction oplog id(s): oplog#717 18/04/26 10:53:10.253 CST P2P message reader for 10.26.23.56(6945):59192 unshared ordered uid=52 dom #1 https://github.com/SnappyDataInc/snappydata/pull/1 INFO snappystore: com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP_SNAPPYDATA INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:314) at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:287) at com.pivotal.gemfirexd.internal.engine.distributed.utils.GemFireXDUtils. throwAssert(GemFireXDUtils.java:1921) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. handleNotDeleted(GfxdIndexManager.java:2749) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. deleteFromIndexes(GfxdIndexManager.java:2490) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.doDelete(GfxdIndexManager.java:2795) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.postEvent(GfxdIndexManager.java:1265) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry. destroy(AbstractRegionEntry.java:905) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroyEntry( AbstractRegionMap.java:4421) at com.gemstone.gemfire.internal.cache.AbstractRegionMap. destroy(AbstractRegionMap.java:1985) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy( LocalRegion.java:8221) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy( LocalRegion.java:8193) at com.gemstone.gemfire.internal.cache.LocalRegionDataView. destroyExistingEntry(LocalRegionDataView.java:87) at com.gemstone.gemfire.internal.cache.TXState.txDestroyExistingEntry( TXState.java:2589) at com.gemstone.gemfire.internal.cache.TXState. destroyExistingEntry(TXState.java:2579) at com.gemstone.gemfire.internal.cache.BucketRegion.destroyAllEntries( BucketRegion.java:885) at com.gemstone.gemfire.internal.cache.BucketRegion. internalCreateAndInsertColumnBatch(BucketRegion.java:828) at com.gemstone.gemfire.internal.cache.BucketRegion. createAndInsertColumnBatch(BucketRegion.java:774) at com.gemstone.gemfire.internal.cache.partitioned. PutAllPRMessage.doLocalPutAll(PutAllPRMessage.java:689) at com.gemstone.gemfire.internal.cache.partitioned.PutAllPRMessage. operateOnPartitionedRegion(PutAllPRMessage.java:343) at com.gemstone.gemfire.internal.cache.partitioned. PartitionMessage.basicProcess(PartitionMessage.java:284) at com.gemstone.gemfire.internal.cache.AbstractOperationMessage.process( AbstractOperationMessage.java:208) at com.gemstone.gemfire.distributed.internal.DistributionMessage. scheduleAction(DistributionMessage.java:431) at com.gemstone.gemfire.distributed.internal.DistributionMessage.schedule( DistributionMessage.java:489) at com.gemstone.gemfire.distributed.internal.DistributionManager. scheduleIncomingMessage(DistributionManager.java:4023) at com.gemstone.gemfire.distributed.internal.DistributionManager. handleIncomingDMsg(DistributionManager.java:3631) at com.gemstone.gemfire.distributed.internal.DistributionManager$ MyListener.messageReceived(DistributionManager.java:4883) at com.gemstone.gemfire.distributed.internal.membership.jgroup. JGroupMembershipManager.processMessage(JGroupMembershipManager.java:2209) at com.gemstone.gemfire.distributed.internal.membership.jgroup. JGroupMembershipManager.handleOrDeferMessage(JGroupMembershipManager.java: 2117) at com.gemstone.gemfire.distributed.internal.membership.jgroup. JGroupMembershipManager$MyDCReceiver.messageReceived( JGroupMembershipManager.java:670) at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.receive( DirectChannel.java:851) at com.gemstone.gemfire.internal.tcp.TCPConduit. messageReceived(TCPConduit.java:874) at com.gemstone.gemfire.internal.tcp.Connection. dispatchMessage(Connection.java:4155) at com.gemstone.gemfire.internal.tcp.Connection.readAndDispatchMessage( Connection.java:4698) at com.gemstone.gemfire.internal.tcp.Connection.processMessage( Connection.java:4579) at com.gemstone.gemfire.internal.tcp.Connection. processNIOStream(Connection.java:4243) at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1813) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

18/04/26 10:53:10.718 CST Pooled High Priority Message Processor 65 INFO snappystore: com.pivotal.gemfirexd. internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 3932725; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.2.6.26(17265):20875 at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:314) at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:287) at com.pivotal.gemfirexd.internal.engine.distributed.utils.GemFireXDUtils. throwAssert(GemFireXDUtils.java:1921) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. handleNotDeleted(GfxdIndexManager.java:2749) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. deleteFromIndexes(GfxdIndexManager.java:2490) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.doDelete(GfxdIndexManager.java:2795) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.postEvent(GfxdIndexManager.java:1265) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry. destroy(AbstractRegionEntry.java:905) at com.gemstone.gemfire.internal.cache.TXState.rollBackUncommittedEntries( TXState.java:1476) at com.gemstone.gemfire.internal.cache.TXState.rollback(TXState.java:881) at com.gemstone.gemfire.internal.cache.TXManagerImpl.rollback( TXManagerImpl.java:1276) at com.gemstone.gemfire.internal.cache.TXStateProxy.rollback( TXStateProxy.java:2428) at com.gemstone.gemfire.internal.cache.TXRemoteRollbackMessage. operateOnTX(TXRemoteRollbackMessage.java:77) at com.gemstone.gemfire.internal.cache.TXMessage.basicProcess( TXMessage.java:86) at com.gemstone.gemfire.internal.cache.AbstractOperationMessage.process( AbstractOperationMessage.java:208) at com.gemstone.gemfire.distributed.internal.DistributionMessage. scheduleAction(DistributionMessage.java:431) at com.gemstone.gemfire.distributed.internal.DistributionMessage$1.run( DistributionMessage.java:503) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at com.gemstone.gemfire.distributed.internal.DistributionManager. runUntilShutdown(DistributionManager.java:728) at com.gemstone.gemfire.distributed.internal.DistributionManager$6$1.run( DistributionManager.java:1058) at java.lang.Thread.run(Thread.java:745)

18/04/26 10:53:16.841 CST OplogCompactor SNAPPY-INTERNAL-DELTA for oplog oplog#717 INFO snappystore: compaction did 294,280 creates and updates in 15,882 ms 18/04/26 10:53:16.853 CST Oplog Delete Task INFO snappystore: Deleted oplog#717 crf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 10:53:16.858 CST Oplog Delete Task INFO snappystore: Deleted oplog#717 krf for disk store SNAPPY-INTERNAL-DELTA.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/1014, or mute the thread https://github.com/notifications/unsubscribe-auth/AALmGSZO9KG7UuETWFkuMhqYpEDJkdLjks5tsTfwgaJpZM4Tkckp .

-- Regards, Suranjan Kumar SnappyData (http://www.snappydata.io)

crabo commented 6 years ago

@suranjan : as the first line in my post: I'm using jdbc sql "insert into ys_send values(item_1..),.....(item_1000,.....)" batch insertion.

just ingest some data with batch insert in SQL. I'm using Snappydata v1.01, JDK8, with 2 nodes both 16CPU/64GB RAM/1T HDD

crabo commented 6 years ago

keep happening all the time:

18/04/26 16:11:34.092 CST P2P message reader for 10.26.23.56(6945):59192 unshared ordered uid=102 dom #2 INFO snappystore: Created oplog#1243 crf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.093 CST P2P message reader for 10.26.23.56(6945):59192 unshared ordered uid=102 dom #2 INFO snappystore: createKrfAsync called for oplog: oplog#1242, parent: SNAPPY-INTERNAL-DELTA 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 krf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 idxkrf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 /data/s1/snappy/server/snappy-internal-delta/BACKUPSNAPPY-INTERNAL-DELTA_1242.1.idxkrf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:43.773 CST ThriftProcessor-12 INFO snappystore: Region /APP/YS_RESP putAll: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. 18/04/26 16:11:43.781 CST ThriftProcessor-12 INFO snappystore: [TRACE 2018/04/26 16:11:43.780 CST GFXD:TRACE tid=0x236] (XID = 949(UserTransaction)@563ed9ce;txState=GfxdTXStateProxy@(TXId[3:77186 => 10.27.6.145(17265):20875],state=OPEN{SNAPSHOT}{COORDINATOR}{LOCAL_TXSTATE@2297539}{DIRTY}{SYNCCOMMITS}@0x5b449565);supportsTX=true), (SESSIONID = 352), (DATABASE = snappydata), (DRDAID = null), Failed Statement is: INSERT INTO ys_resp(BatchNo,IsDELIVRD,SendStatus,ErrorInfo,ReportTime,ReportDate,ReportDateHour,ChannelNumber,RPmark1,RPmark2,LastUpdateOn) VALUES (?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?) with 11000 parameters ;value=426414424643320662^0,type=0;......

ERROR 38000: The exception 'com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. ' was thrown while evaluating an expression. at com.pivotal.gemfirexd.internal.iapi.error.StandardException.newException(StandardException.java:473) at com.pivotal.gemfirexd.internal.engine.Misc.processGemFireException(Misc.java:1046) at com.pivotal.gemfirexd.internal.engine.Misc.processFunctionException(Misc.java:831) at com.pivotal.gemfirexd.internal.engine.Misc.processFunctionException(Misc.java:784) at com.pivotal.gemfirexd.internal.engine.Misc.processKnownGemFireException(Misc.java:1007) at com.pivotal.gemfirexd.internal.engine.Misc.processGemFireException(Misc.java:1018) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer.insertMultipleRows(GemFireContainer.java:3604) at com.pivotal.gemfirexd.internal.engine.sql.execute.GemFireInsertResultSet.handleMultipleInserts(GemFireInsertResultSet.java:306) at com.pivotal.gemfirexd.internal.engine.sql.execute.GemFireInsertResultSet.open(GemFireInsertResultSet.java:210) at com.pivotal.gemfirexd.internal.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:593) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:2175) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1984) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedCallableStatement.executeStatement(EmbedCallableStatement.java:133) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:393) at io.snappydata.thrift.server.SnappyDataServiceImpl.executePreparedUpdate(SnappyDataServiceImpl.java:2495) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate.getResult(SnappyDataService.java:1631) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate.getResult(SnappyDataService.java:1615) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at io.snappydata.thrift.server.SnappyDataServiceImpl$Processor.process(SnappyDataServiceImpl.java:201) at io.snappydata.thrift.server.SnappyThriftServerThreadPool$WorkerProcess.run(SnappyThriftServerThreadPool.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at io.snappydata.thrift.server.SnappyThriftServer$1.lambda$newThread$0(SnappyThriftServer.java:143) at java.lang.Thread.run(Thread.java:745) ============= begin nested exception, level (1) =========== com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2YS_RESPSNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend(PartitionedRegion.java:2498) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.postPutAll(LocalRegionDataView.java:406) at com.gemstone.gemfire.internal.cache.TXStateProxy.postPutAll(TXStateProxy.java:3843) at com.gemstone.gemfire.internal.cache.LocalRegion.basicPutAll(LocalRegion.java:11846) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer.insertMultipleRows(GemFireContainer.java:3539) ... 17 more ============= end nested exception, level (1) =========== ============= begin nested exception, level (2) =========== com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESPSNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 at com.pivotal.gemfirexd.internal.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:314) at com.pivotal.gemfirexd.internal.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:287) at com.pivotal.gemfirexd.internal.engine.distributed.utils.GemFireXDUtils.throwAssert(GemFireXDUtils.java:1921) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.handleNotDeleted(GfxdIndexManager.java:2749) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.deleteFromIndexes(GfxdIndexManager.java:2490) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.doDelete(GfxdIndexManager.java:2795) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager.postEvent(GfxdIndexManager.java:1265) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry.destroy(AbstractRegionEntry.java:905) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroyEntry(AbstractRegionMap.java:4421) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1985) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:8221) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:8193) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:87) at com.gemstone.gemfire.internal.cache.TXStateProxy$17.operateOnSharedDataView(TXStateProxy.java:4540) at com.gemstone.gemfire.internal.cache.TXStateProxy.performOp(TXStateProxy.java:3041) at com.gemstone.gemfire.internal.cache.TXStateProxy.destroyExistingEntry(TXStateProxy.java:1742) at com.gemstone.gemfire.internal.cache.BucketRegion.destroyAllEntries(BucketRegion.java:885) at com.gemstone.gemfire.internal.cache.BucketRegion.internalCreateAndInsertColumnBatch(BucketRegion.java:828) at com.gemstone.gemfire.internal.cache.BucketRegion.createAndInsertColumnBatch(BucketRegion.java:774) at com.gemstone.gemfire.internal.cache.partitioned.PutAllPRMessage.doLocalPutAll(PutAllPRMessage.java:689) at com.gemstone.gemfire.internal.cache.PartitionedRegion.tryToSendOnePutAllMessage(PartitionedRegion.java:3045) at com.gemstone.gemfire.internal.cache.PartitionedRegion.sendOrWaitForRespWithRetry(PartitionedRegion.java:2872) at com.gemstone.gemfire.internal.cache.PartitionedRegion.sendMsgByBucket(PartitionedRegion.java:2757) at com.gemstone.gemfire.internal.cache.PartitionedRegion.access$100(PartitionedRegion.java:239) at com.gemstone.gemfire.internal.cache.PartitionedRegion$8.execute(PartitionedRegion.java:2423) at com.gemstone.gnu.trove.THashMap.forEachEntry(THashMap.java:309) at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend(PartitionedRegion.java:2415) ... 21 more ============= end nested exception, level (2) ===========

18/04/26 16:12:06.687 CST ThriftProcessor-10 INFO snappystore: Created oplog#1244 drf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:12:06.690 CST ThriftProcessor-10 INFO snappystore: Created oplog#1244 crf for disk store SNAPPY-INTERNAL-DELTA.

suranjan commented 6 years ago

Could you please tell me the version of snappydata that you are using?

On Fri, Apr 27, 2018 at 8:10 AM, crabo notifications@github.com wrote:

keep happening all the time:

18/04/26 16:11:34.092 CST P2P message reader for 10.26.23.56(6945):59192 unshared ordered uid=102 dom #2 https://github.com/SnappyDataInc/snappydata/pull/2 INFO snappystore: Created oplog#1243 crf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.093 CST P2P message reader for 10.26.23.56(6945):59192 unshared ordered uid=102 dom #2 https://github.com/SnappyDataInc/snappydata/pull/2 INFO snappystore: createKrfAsync called for oplog: oplog#1242, parent: SNAPPY-INTERNAL-DELTA 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 krf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 idxkrf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:34.949 CST Idle OplogCompactor INFO snappystore: Created oplog#1242 /data/s1/snappy/server/snappy- internal-delta/BACKUPSNAPPY-INTERNAL-DELTA_1242.1.idxkrf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:11:43.773 CST ThriftProcessor-12 INFO snappystore: Region /APP/YS_RESP putAll: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd. internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. 18/04/26 16:11:43.781 CST ThriftProcessor-12 INFO snappystore: [TRACE 2018/04/26 16:11:43.780 CST GFXD:TRACE tid=0x236] (XID = 949(UserTransaction)@563ed9ce;txState=GfxdTXStateProxy@(TXId[3:77186 => 10.27.6.145(17265):20875],state=OPEN{SNAPSHOT}{COORDINATOR}{LOCAL_TXSTATE@ 2297539}{DIRTY}{SYNCCOMMITS}@0x5b449565);supportsTX=true), (SESSIONID = 352), (DATABASE = snappydata), (DRDAID = null), Failed Statement is: INSERT INTO ys_resp(BatchNo,IsDELIVRD,SendStatus,ErrorInfo,ReportTime,ReportDate, ReportDateHour,ChannelNumber,RPmark1,RPmark2,LastUpdateOn) VALUES (?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?) with 11000 parameters ;value=426414424643320662^0,type=0;......

ERROR 38000: The exception 'com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd. internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2YS_RESP__SNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. ' was thrown while evaluating an expression. at com.pivotal.gemfirexd.internal.iapi.error.StandardException. newException(StandardException.java:473) at com.pivotal.gemfirexd.internal.engine.Misc. processGemFireException(Misc.java:1046) at com.pivotal.gemfirexd.internal.engine.Misc. processFunctionException(Misc.java:831) at com.pivotal.gemfirexd.internal.engine.Misc. processFunctionException(Misc.java:784) at com.pivotal.gemfirexd.internal.engine.Misc. processKnownGemFireException(Misc.java:1007) at com.pivotal.gemfirexd.internal.engine.Misc. processGemFireException(Misc.java:1018) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer. insertMultipleRows(GemFireContainer.java:3604) at com.pivotal.gemfirexd.internal.engine.sql.execute. GemFireInsertResultSet.handleMultipleInserts(GemFireInsertResultSet.java: 306) at com.pivotal.gemfirexd.internal.engine.sql.execute. GemFireInsertResultSet.open(GemFireInsertResultSet.java:210) at com.pivotal.gemfirexd.internal.impl.sql.GenericPreparedStatement. execute(GenericPreparedStatement.java:593) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedStatement. executeStatement(EmbedStatement.java:2175) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement. executeStatement(EmbedPreparedStatement.java:1984) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedCallableStatement. executeStatement(EmbedCallableStatement.java:133) at com.pivotal.gemfirexd.internal.impl.jdbc.EmbedPreparedStatement. executeUpdate(EmbedPreparedStatement.java:393) at io.snappydata.thrift.server.SnappyDataServiceImpl. executePreparedUpdate(SnappyDataServiceImpl.java:2495) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate. getResult(SnappyDataService.java:1631) at io.snappydata.thrift.SnappyDataService$Processor$executePreparedUpdate. getResult(SnappyDataService.java:1615) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at io.snappydata.thrift.server.SnappyDataServiceImpl$Processor.process( SnappyDataServiceImpl.java:201) at io.snappydata.thrift.server.SnappyThriftServerThreadPool$ WorkerProcess.run(SnappyThriftServerThreadPool.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at io.snappydata.thrift.server.SnappyThriftServer$1.lambda$newThread$0( SnappyThriftServer.java:143) at java.lang.Thread.run(Thread.java:745) ============= begin nested exception, level (1) =========== com.gemstone.gemfire.internal.cache.PutAllPartialResultException: Key tableinfo(true).CompactCompositeRegionKey@19eb0bbb=(4297781468) and possibly others failed to put due to com.pivotal.gemfirexd. internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2YS_RESPSNAPPYDATA_INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 The putAll operation failed to put 29 out of 1000 entries. at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend( PartitionedRegion.java:2498) at com.gemstone.gemfire.internal.cache.LocalRegionDataView.postPutAll( LocalRegionDataView.java:406) at com.gemstone.gemfire.internal.cache.TXStateProxy.postPutAll( TXStateProxy.java:3843) at com.gemstone.gemfire.internal.cache.LocalRegion.basicPutAll( LocalRegion.java:11846) at com.pivotal.gemfirexd.internal.engine.store.GemFireContainer. insertMultipleRows(GemFireContainer.java:3539) ... 17 more ============= end nested exception, level (1) =========== ============= begin nested exception, level (2) =========== com.pivotal.gemfirexd.internal.shared.common.sanity.AssertFailure: ASSERT FAILED Delete did not happen on index: APP.2__YS_RESPSNAPPYDATA_ INTERNAL_ROWID:base-table:APP.YS_RESP for indexKey: 2761374; with RowLocation: null; indexedColumns: [12]; unique=true; myID: 10.27.6.145(17265):20875 at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:314) at com.pivotal.gemfirexd.internal.shared.common.sanity. SanityManager.THROWASSERT(SanityManager.java:287) at com.pivotal.gemfirexd.internal.engine.distributed.utils.GemFireXDUtils. throwAssert(GemFireXDUtils.java:1921) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. handleNotDeleted(GfxdIndexManager.java:2749) at com.pivotal.gemfirexd.internal.engine.access.index.GfxdIndexManager. deleteFromIndexes(GfxdIndexManager.java:2490) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.doDelete(GfxdIndexManager.java:2795) at com.pivotal.gemfirexd.internal.engine.access.index. GfxdIndexManager.postEvent(GfxdIndexManager.java:1265) at com.gemstone.gemfire.internal.cache.AbstractRegionEntry. destroy(AbstractRegionEntry.java:905) at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroyEntry( AbstractRegionMap.java:4421) at com.gemstone.gemfire.internal.cache.AbstractRegionMap. destroy(AbstractRegionMap.java:1985) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy( LocalRegion.java:8221) at com.gemstone.gemfire.internal.cache.LocalRegion.mapDestroy( LocalRegion.java:8193) at com.gemstone.gemfire.internal.cache.LocalRegionDataView. destroyExistingEntry(LocalRegionDataView.java:87) at com.gemstone.gemfire.internal.cache.TXStateProxy$17. operateOnSharedDataView(TXStateProxy.java:4540) at com.gemstone.gemfire.internal.cache.TXStateProxy.performOp( TXStateProxy.java:3041) at com.gemstone.gemfire.internal.cache.TXStateProxy.destroyExistingEntry( TXStateProxy.java:1742) at com.gemstone.gemfire.internal.cache.BucketRegion.destroyAllEntries( BucketRegion.java:885) at com.gemstone.gemfire.internal.cache.BucketRegion. internalCreateAndInsertColumnBatch(BucketRegion.java:828) at com.gemstone.gemfire.internal.cache.BucketRegion. createAndInsertColumnBatch(BucketRegion.java:774) at com.gemstone.gemfire.internal.cache.partitioned. PutAllPRMessage.doLocalPutAll(PutAllPRMessage.java:689) at com.gemstone.gemfire.internal.cache.PartitionedRegion. tryToSendOnePutAllMessage(PartitionedRegion.java:3045) at com.gemstone.gemfire.internal.cache.PartitionedRegion. sendOrWaitForRespWithRetry(PartitionedRegion.java:2872) at com.gemstone.gemfire.internal.cache.PartitionedRegion.sendMsgByBucket( PartitionedRegion.java:2757) at com.gemstone.gemfire.internal.cache.PartitionedRegion. access$100(PartitionedRegion.java:239) at com.gemstone.gemfire.internal.cache.PartitionedRegion$8. execute(PartitionedRegion.java:2423) at com.gemstone.gnu.trove.THashMap.forEachEntry(THashMap.java:309) at com.gemstone.gemfire.internal.cache.PartitionedRegion.postPutAllSend( PartitionedRegion.java:2415) ... 21 more ============= end nested exception, level (2) ===========

18/04/26 16:12:06.687 CST ThriftProcessor-10 INFO snappystore: Created oplog#1244 drf for disk store SNAPPY-INTERNAL-DELTA. 18/04/26 16:12:06.690 CST ThriftProcessor-10 INFO snappystore: Created oplog#1244 crf for disk store SNAPPY-INTERNAL-DELTA.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SnappyDataInc/snappydata/issues/1014#issuecomment-384846808, or mute the thread https://github.com/notifications/unsubscribe-auth/AALmGRJPmvjbU48wkx9bg9oZMfxs8PWqks5tsoUigaJpZM4Tkckp .

-- Regards, Suranjan Kumar SnappyData (http://www.snappydata.io)

crabo commented 6 years ago

@suranjan : I've already post in the 2nd post.

"just ingest some data with batch insert in SQL. I'm using Snappydata v1.01, JDK8, with 2 nodes both 16CPU/64GB RAM/1T HDD"

pls read in a little more detail.