java.lang.IllegalStateException: Triyng to access a destroyed ref.

[uberbrodt]

I'm seeing this error after running a process with about 25-100 reads/writes a minute for around 15-30 minutes. The line that seems to be blowing up is this:

transactional { val result = select[SocialMediaStatus] where(_.socialMediaId :== status.getId, _.socialNetwork :== SocialNetwork.Twitter) ......

Without getting too detailed, status is a Twitter4j.Status object, so this query is basically checking whether or not we've already stored the status. Here's the error I'm getting from that query, which only happens intermittently

java.lang.IllegalStateException: Triyng to access a destroyed ref. at net.fwbrasil.radon.transaction.RefSnapshooter.validateIfDestroyed(RefSnapshooter.scala:60) at net.fwbrasil.radon.transaction.RefSnapshooter.getSnapshot(RefSnapshooter.scala:36) at net.fwbrasil.radon.transaction.RefSnapshooter.getSnapshot(RefSnapshooter.scala:22) at net.fwbrasil.radon.transaction.RefSnapshooter.snapshotRead(RefSnapshooter.scala:41) at net.fwbrasil.radon.transaction.Transaction.get(Transaction.scala:53) at net.fwbrasil.radon.ref.Ref.get(Ref.scala:109) at net.fwbrasil.activate.entity.Var.net$fwbrasil$activate$entity$Var$$super$get(Var.scala:63) at net.fwbrasil.activate.entity.Var$$anonfun$get$1.apply(Var.scala:63) at net.fwbrasil.activate.entity.Var$$anonfun$get$1.apply(Var.scala:60) at net.fwbrasil.activate.entity.Var.doInitialized(Var.scala:102) at net.fwbrasil.activate.entity.Var.get(Var.scala:60) at net.fwbrasil.activate.entity.Var.getValue(Var.scala:68) at net.fwbrasil.activate.cache.live.LiveCache.entityProperty(LiveCache.scala:494) at net.fwbrasil.activate.cache.live.LiveCache.entityPropertyPathRef(LiveCache.scala:477) at net.fwbrasil.activate.cache.live.LiveCache.executeStatementEntitySourceValue(LiveCache.scala:466) at net.fwbrasil.activate.cache.live.LiveCache.executeStatementEntityValue(LiveCache.scala:459) at net.fwbrasil.activate.cache.live.LiveCache.executeStatementSelectValue(LiveCache.scala:449) at net.fwbrasil.activate.cache.live.LiveCache.executeStatementValue(LiveCache.scala:431) at net.fwbrasil.activate.cache.live.LiveCache.executeCompositeOperatorCriteria(LiveCache.scala:348) at net.fwbrasil.activate.cache.live.LiveCache.executeCriteria(LiveCache.scala:344) at net.fwbrasil.activate.cache.live.LiveCache.executeStatementBooleanValue(LiveCache.scala:421) at net.fwbrasil.activate.cache.live.LiveCache.executeBooleanOperatorCriteria(LiveCache.scala:411) at net.fwbrasil.activate.cache.live.LiveCache.executeCriteria(LiveCache.scala:340) at net.fwbrasil.activate.cache.live.LiveCache.executeQueryWithEntitySourcesMap(LiveCache.scala:297) at net.fwbrasil.activate.cache.live.LiveCache$$anonfun$executeQueryWithEntitySources$1.apply(LiveCache.scala:278) at net.fwbrasil.activate.cache.live.LiveCache$$anonfun$executeQueryWithEntitySources$1.apply(LiveCache.scala:277) at scala.collection.immutable.List.foreach(List.scala:318) at net.fwbrasil.activate.cache.live.LiveCache.executeQueryWithEntitySources(LiveCache.scala:277) at net.fwbrasil.activate.cache.live.LiveCache.entitiesFromCache(LiveCache.scala:159) at net.fwbrasil.activate.cache.live.LiveCache.executeQuery(LiveCache.scala:184) at net.fwbrasil.activate.statement.query.QueryContext$$anonfun$1.apply(QueryContext.scala:29) at net.fwbrasil.activate.statement.query.QueryContext$$anonfun$1.apply(QueryContext.scala:28) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at net.fwbrasil.activate.statement.query.QueryContext$class.executeQuery(QueryContext.scala:28) ...

I can try and setup an example project that replicates the issue, but wanted to get this in front of you and see if there was a quick explanation or fix first.

[fwbrasil]

Could you test if the problem occurs using the 1.4-SNAPSHOT in my repo? (http://fwbrasil.net/maven/net/fwbrasil/)

[uberbrodt]

Thanks for the quick reply. I pulled the snapshots a couple of hours ago, and it seems the problem is still there (perhaps worse, as it seems to no longer be intermittent but will lock up after a few seconds, continually spitting this error out).

23:17:28.691 TKD [TwitterSystem-akka.actor.default-dispatcher-5] DEBUG TwitterLogger - Matching Compound Terms 23:17:28.715 TKD [TwitterSystem-akka.actor.default-dispatcher-7] ERROR e.s.twitter.ProcessTweetActor - Triyng to access a destroyed ref. java.lang.IllegalStateException: Triyng to access a destroyed ref. at net.fwbrasil.radon.transaction.RefSnapshooter.validateIfDestroyed(RefSnapshooter.scala:60) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.radon.transaction.RefSnapshooter.getSnapshot(RefSnapshooter.scala:36) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.radon.transaction.RefSnapshooter.getSnapshot(RefSnapshooter.scala:22) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.radon.transaction.RefSnapshooter.snapshotRead(RefSnapshooter.scala:41) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.radon.transaction.Transaction.get(Transaction.scala:53) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.radon.ref.Ref.get(Ref.scala:109) ~[radon-stm_2.10-1.3.jar:1.3] at net.fwbrasil.activate.entity.Var.net$fwbrasil$activate$entity$Var$$super$get(Var.scala:63) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.entity.Var$$anonfun$get$1.apply(Var.scala:63) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.entity.Var$$anonfun$get$1.apply(Var.scala:60) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.entity.Var.doInitialized(Var.scala:102) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.entity.Var.get(Var.scala:60) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.entity.Var.getValue(Var.scala:68) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.entityProperty(LiveCache.scala:494) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.entityPropertyPathRef(LiveCache.scala:477) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeStatementEntitySourceValue(LiveCache.scala:466) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeStatementEntityValue(LiveCache.scala:459) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeStatementSelectValue(LiveCache.scala:449) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeStatementValue(LiveCache.scala:431) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeCompositeOperatorCriteria(LiveCache.scala:348) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeCriteria(LiveCache.scala:344) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeStatementBooleanValue(LiveCache.scala:421) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeBooleanOperatorCriteria(LiveCache.scala:411) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeCriteria(LiveCache.scala:340) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeQueryWithEntitySourcesMap(LiveCache.scala:297) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache$$anonfun$executeQueryWithEntitySources$1.apply(LiveCache.scala:278) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache$$anonfun$executeQueryWithEntitySources$1.apply(LiveCache.scala:277) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at scala.collection.immutable.List.foreach(List.scala:318) ~[scala-library.jar:0.12.3] at net.fwbrasil.activate.cache.live.LiveCache.executeQueryWithEntitySources(LiveCache.scala:277) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.entitiesFromCache(LiveCache.scala:159) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.cache.live.LiveCache.executeQuery(LiveCache.scala:184) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.statement.query.QueryContext$$anonfun$1.apply(QueryContext.scala:30) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.statement.query.QueryContext$$anonfun$1.apply(QueryContext.scala:29) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library.jar:0.12.3] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[scala-library.jar:0.12.3] at scala.collection.immutable.List.foreach(List.scala:318) ~[scala-library.jar:0.12.3] at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[scala-library.jar:0.12.3] at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[scala-library.jar:0.12.3] at net.fwbrasil.activate.statement.query.QueryContext$class.executeQuery(QueryContext.scala:29) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at eventhash.common.persistence.EventhashMongoContext$.executeQuery(EventhashMongoContext.scala:14) ~[classes/:na] at net.fwbrasil.activate.statement.query.Query.execute(Query.scala:30) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.statement.query.QueryContext$class.net$fwbrasil$activate$statement$query$QueryContext$$_allWhere(QueryContext.scala:224) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at net.fwbrasil.activate.statement.query.QueryContext$SelectEntity.where(QueryContext.scala:233) ~[activate-core_2.10-1.4-SNAPSHOT.jar:1.4-SNAPSHOT] at eventhash.socialmediaclients.twitter.ProcessTweetActor$$anonfun$process$1.apply$mcV$sp(ProcessTweetActor.scala:26) ~[classes/:na]

[uberbrodt]

I tested again with 1.3.1 to verify that it mostly works, with the occasional "destroyed ref" error and almost completely locks up in 1.4-SNAPSHOT. Looking at your commit, looks like you removed the isDeletedSnapshot check in the retrieve-from-cache call, so that probably makes sense.

[fwbrasil]

My fault, I forgot to update the activate dependency to radon-stm 1.4-SNAPSHOT. Could you update the dependencies and try again? Please make sure that radon-stm is in the right version. Sorry for the inconvenience.

There was an inconsistency with isDeletedSpanshot that I solved by removing its call and validating access to destroyed refs at commit time. Take a look on these commits in radom-stm:

https://github.com/fwbrasil/radon-stm/commit/a4f81b906ea03c99de0a2bd5a91bc70a1080f73b https://github.com/fwbrasil/radon-stm/commit/ffe71d73e727baec2bec9728318090a7ed3a1ac9

There are some edges to solve, but I want to know if the general solution solves your case.

[uberbrodt]

Perfeito! That seems to have done the trick. Thanks for your help

[fwbrasil]

Nice! I will finish some modifications and make a release with this fix in the next days.

[fwbrasil]

The release 1.3.2 is available at maven central with this fix.