search

JanusGraph / janusgraph

JanusGraph: an open-source, distributed graph database
https://janusgraph.org
Other
5.32k stars 1.17k forks source link

Permanent failure in storage backend CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:315) #1526

Open madhuvasu opened 5 years ago

madhuvasu commented 5 years ago

We are seeing the following stacktrace when trying to ingest data on JanusGraph 0.2.0. We have cassandra, elasticsearch and janusgraph run on the same local system.

org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
    at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
    at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:95)
    at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:143)
    at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:200)
    at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134)
    at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:737)
    at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1374)
    at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.acceptRecord(VertexLoaderWorker.java:109)
    at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.access$000(VertexLoaderWorker.java:35)
    at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker$1.accept(VertexLoaderWorker.java:130)
    at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker$1.accept(VertexLoaderWorker.java:126)
    at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:899)
    at com.ibm.janusgraph.utils.importer.vertex.VertexLoaderWorker.run(VertexLoaderWorker.java:126)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend
    at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftKeyColumnValueStore.convertException(CassandraThriftKeyColumnValueStore.java:263)
    at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:315)
    at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:79)
    at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:98)
    at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:95)
    at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
    at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
    ... 17 more
Caused by: TimedOutException(acknowledged_by:0, acknowledged_by_batchlog:true)
    at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29624)
    at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29592)
    at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result.read(Cassandra.java:29526)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_atomic_batch_mutate(Cassandra.java:1108)
    at org.apache.cassandra.thrift.Cassandra$Client.atomic_batch_mutate(Cassandra.java:1094)
    at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.mutateMany(CassandraThriftStoreManager.java:310)
    ... 22 more

Please find attached the janusgraph configuration.

janusgraph-fci-graph.properties.txt

farodin91 commented 5 years ago

Do you get the same issue with janusgraph 0.3.1?

madhuvasu commented 5 years ago

Yes, we are seeing this exception on JanusGraph 0.3.1 as well.

porunov commented 4 years ago

@madhuvasu Thrift support was deprecated in 0.4.1 version of JanusGraph. Thrift is going to be completely removed in 0.6.0 version of JanusGraph due to deprecation. The PR is #1800

Are you able to migrate to CQL storage backend instead of deprecated Thrift backend? Thrift will be removed in Cassandra 4.