Airport loading deadlock

[patsonluk]

Due to lazy loading of income and campaigns which airports can have cyclic references:

Found one Java-level deadlock:

"application-akka.actor.default-dispatcher-2": waiting to lock monitor 0x00007fd7ea60e700 (object 0x00000006c36ac070, a com.patson.model.Airport), which is held by "application-akka.actor.default-dispatcher-16" "application-akka.actor.default-dispatcher-16": waiting to lock monitor 0x00007fd79c4a1400 (object 0x00000006c3663278, a com.patson.model.Airport), which is held by "application-akka.actor.default-dispatcher-2"

Java stack information for the threads listed above:

"application-akka.actor.default-dispatcher-2": at com.patson.model.Airport.income$lzycompute(Airport.scala:61)

• waiting to lock <0x00000006c36ac070> (a com.patson.model.Airport) at com.patson.model.Airport.income(Airport.scala:61) at com.patson.model.CampaignDelegateTask.(BusyDelegate.scala:34) at com.patson.model.DelegateTask$.$anonfun$campaign$1(BusyDelegate.scala:18) at com.patson.model.DelegateTask$.$anonfun$campaign$1$adapted(BusyDelegate.scala:18) at com.patson.model.DelegateTask$$$Lambda$1970/0x0000000840c13840.apply(Unknown Source) at com.patson.data.DelegateSource$.loadBusyDelegatesByCampaigns(DelegateSource.scala:213) at com.patson.data.AirportSource$.getCampaignBonuses(AirportSource.scala:86) at com.patson.data.AirportSource$.loadAirportsByQueryString(AirportSource.scala:323) at com.patson.data.AirportSource$.loadAirportsByCriteria(AirportSource.scala:42) at com.patson.data.AirportSource$.loadAirportById(AirportSource.scala:519) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:48) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:46) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3451) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2200) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
• locked <0x00000006de5802f0> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) at com.google.common.cache.LocalCache.get(LocalCache.java:3855) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3878) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4780) at com.patson.util.AirportCache$.getAirport(AirportCache.scala:18) at com.patson.model.AirportRating$.rateAirport(AirportRating.scala:34) at com.patson.model.Airport.rating$lzycompute(Airport.scala:91)
• locked <0x00000006c3663278> (a com.patson.model.Airport) at com.patson.model.Airport.rating(Airport.scala:91) at com.patson.model.AirlineBase.getUpkeep$lzycompute(AirlineBase.scala:24)
• locked <0x00000006de580330> (a com.patson.model.AirlineBase) at com.patson.model.AirlineBase.getUpkeep(AirlineBase.scala:22) at controllers.package$AirlineBaseFormat$.writes(package.scala:318) at controllers.package$AirlineBaseFormat$.writes(package.scala:297) at play.api.libs.json.LowPriorityWrites.$anonfun$traversableWrites$2(Writes.scala:334) at play.api.libs.json.LowPriorityWrites$$Lambda$1585/0x0000000840ac7040.apply(Unknown Source) at scala.collection.immutable.List.foreach(List.scala:312) at play.api.libs.json.LowPriorityWrites.$anonfun$traversableWrites$1(Writes.scala:333) at play.api.libs.json.LowPriorityWrites$$Lambda$1584/0x0000000840ac6840.apply(Unknown Source) at play.api.libs.json.Writes$$anon$6.writes(Writes.scala:157) at play.api.libs.json.Json$.toJson(Json.scala:187) at controllers.AirlineApplication.$anonfun$getAirline$1(AirlineApplication.scala:174) at controllers.AirlineApplication$$Lambda$1790/0x0000000840b6b840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.mvc.Security$AuthenticatedBuilder.$anonfun$authenticate$1(Security.scala:199) at play.api.mvc.Security$AuthenticatedBuilder$$Lambda$1612/0x0000000840ade840.apply(Unknown Source) at scala.Option.map(Option.scala:243) at play.api.mvc.Security$AuthenticatedBuilder.authenticate(Security.scala:198) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:191) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:181) at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426) at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98) at play.api.mvc.Action$$Lambda$1326/0x0000000840972040.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1331/0x0000000840974c40.apply(Unknown Source) at scala.util.Try$.apply(Try.scala:212) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1327/0x0000000840972840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:219) at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441) at play.core.server.AkkaHttpServer$$Lambda$1318/0x000000084096c840.apply(Unknown Source) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51) at akka.http.scaladsl.util.FastFuture$$$Lambda$1319/0x000000084096d840.apply(Unknown Source) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$407/0x0000000840488440.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) "application-akka.actor.default-dispatcher-16": at com.patson.model.Airport.income$lzycompute(Airport.scala:61)
• waiting to lock <0x00000006c3663278> (a com.patson.model.Airport) at com.patson.model.Airport.income(Airport.scala:61) at com.patson.model.CampaignDelegateTask.(BusyDelegate.scala:34) at com.patson.model.DelegateTask$.$anonfun$campaign$1(BusyDelegate.scala:18) at com.patson.model.DelegateTask$.$anonfun$campaign$1$adapted(BusyDelegate.scala:18) at com.patson.model.DelegateTask$$$Lambda$1970/0x0000000840c13840.apply(Unknown Source) at com.patson.data.DelegateSource$.loadBusyDelegatesByCampaigns(DelegateSource.scala:213) at com.patson.data.AirportSource$.getCampaignBonuses(AirportSource.scala:86) at com.patson.data.AirportSource$.loadAirportsByQueryString(AirportSource.scala:323) at com.patson.data.AirportSource$.loadAirportsByCriteria(AirportSource.scala:42) at com.patson.data.AirportSource$.loadAirportById(AirportSource.scala:519) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:48) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:46) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3451) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2200) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
• locked <0x00000006de600580> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) at com.google.common.cache.LocalCache.get(LocalCache.java:3855) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3878) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4780) at com.patson.util.AirportCache$.getAirport(AirportCache.scala:18) at com.patson.model.AirportRating$.rateAirport(AirportRating.scala:34) at com.patson.model.Airport.rating$lzycompute(Airport.scala:91)
• locked <0x00000006c36ac070> (a com.patson.model.Airport) at com.patson.model.Airport.rating(Airport.scala:91) at com.patson.model.AirlineBase.getUpkeep$lzycompute(AirlineBase.scala:24)
• locked <0x00000006de6005c0> (a com.patson.model.AirlineBase) at com.patson.model.AirlineBase.getUpkeep(AirlineBase.scala:22) at controllers.package$AirlineBaseFormat$.writes(package.scala:318) at controllers.package$AirlineBaseFormat$.writes(package.scala:297) at play.api.libs.json.Json$.toJson(Json.scala:187) at controllers.AirlineApplication.$anonfun$getAirline$2(AirlineApplication.scala:170) at controllers.AirlineApplication.$anonfun$getAirline$2$adapted(AirlineApplication.scala:169) at controllers.AirlineApplication$$Lambda$1795/0x0000000840b6e840.apply(Unknown Source) at scala.Option.foreach(Option.scala:438) at controllers.AirlineApplication.$anonfun$getAirline$1(AirlineApplication.scala:169) at controllers.AirlineApplication$$Lambda$1790/0x0000000840b6b840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.mvc.Security$AuthenticatedBuilder.$anonfun$authenticate$1(Security.scala:199) at play.api.mvc.Security$AuthenticatedBuilder$$Lambda$1612/0x0000000840ade840.apply(Unknown Source) at scala.Option.map(Option.scala:243) at play.api.mvc.Security$AuthenticatedBuilder.authenticate(Security.scala:198) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:191) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:181) at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426) at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98) at play.api.mvc.Action$$Lambda$1326/0x0000000840972040.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1331/0x0000000840974c40.apply(Unknown Source) at scala.util.Try$.apply(Try.scala:212) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1327/0x0000000840972840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:219) at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441) at play.core.server.AkkaHttpServer$$Lambda$1318/0x000000084096c840.apply(Unknown Source) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51) at akka.http.scaladsl.util.FastFuture$$$Lambda$1319/0x000000084096d840.apply(Unknown Source) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$407/0x0000000840488440.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Found one Java-level deadlock:

"application-akka.actor.default-dispatcher-3": waiting to lock monitor 0x00007fd7ac8e2700 (object 0x00000006d23a5158, a com.patson.model.Airport), which is held by "application-akka.actor.default-dispatcher-18" "application-akka.actor.default-dispatcher-18": waiting to lock monitor 0x00007fd7982a8700 (object 0x00000006cdffecb8, a com.patson.model.Airport), which is held by "application-akka.actor.default-dispatcher-3"

Java stack information for the threads listed above:

"application-akka.actor.default-dispatcher-3": at com.patson.model.Airport.income$lzycompute(Airport.scala:61)

• waiting to lock <0x00000006d23a5158> (a com.patson.model.Airport) at com.patson.model.Airport.income(Airport.scala:61) at com.patson.model.CampaignDelegateTask.(BusyDelegate.scala:34) at com.patson.model.DelegateTask$.$anonfun$campaign$1(BusyDelegate.scala:18) at com.patson.model.DelegateTask$.$anonfun$campaign$1$adapted(BusyDelegate.scala:18) at com.patson.model.DelegateTask$$$Lambda$1970/0x0000000840c13840.apply(Unknown Source) at com.patson.data.DelegateSource$.loadBusyDelegatesByCampaigns(DelegateSource.scala:213) at com.patson.data.AirportSource$.getCampaignBonuses(AirportSource.scala:86) at com.patson.data.AirportSource$.loadAirportsByQueryString(AirportSource.scala:323) at com.patson.data.AirportSource$.loadAirportsByCriteria(AirportSource.scala:42) at com.patson.data.AirportSource$.loadAirportById(AirportSource.scala:519) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:48) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:46) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3451) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2200) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
• locked <0x00000006de5808c8> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) at com.google.common.cache.LocalCache.get(LocalCache.java:3855) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3878) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4780) at com.patson.util.AirportCache$.getAirport(AirportCache.scala:18) at com.patson.model.AirportRating$.rateAirport(AirportRating.scala:34) at com.patson.model.Airport.rating$lzycompute(Airport.scala:91)
• locked <0x00000006cdffecb8> (a com.patson.model.Airport) at com.patson.model.Airport.rating(Airport.scala:91) at com.patson.model.AirlineBase.getUpkeep$lzycompute(AirlineBase.scala:24)
• locked <0x00000006de580908> (a com.patson.model.AirlineBase) at com.patson.model.AirlineBase.getUpkeep(AirlineBase.scala:22) at controllers.package$AirlineBaseFormat$.writes(package.scala:318) at controllers.package$AirlineBaseFormat$.writes(package.scala:297) at play.api.libs.json.Json$.toJson(Json.scala:187) at controllers.AirlineApplication.$anonfun$getAirline$2(AirlineApplication.scala:170) at controllers.AirlineApplication.$anonfun$getAirline$2$adapted(AirlineApplication.scala:169) at controllers.AirlineApplication$$Lambda$1795/0x0000000840b6e840.apply(Unknown Source) at scala.Option.foreach(Option.scala:438) at controllers.AirlineApplication.$anonfun$getAirline$1(AirlineApplication.scala:169) at controllers.AirlineApplication$$Lambda$1790/0x0000000840b6b840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.mvc.Security$AuthenticatedBuilder.$anonfun$authenticate$1(Security.scala:199) at play.api.mvc.Security$AuthenticatedBuilder$$Lambda$1612/0x0000000840ade840.apply(Unknown Source) at scala.Option.map(Option.scala:243) at play.api.mvc.Security$AuthenticatedBuilder.authenticate(Security.scala:198) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:191) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:181) at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426) at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98) at play.api.mvc.Action$$Lambda$1326/0x0000000840972040.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1331/0x0000000840974c40.apply(Unknown Source) at scala.util.Try$.apply(Try.scala:212) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1327/0x0000000840972840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:219) at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441) at play.core.server.AkkaHttpServer$$Lambda$1318/0x000000084096c840.apply(Unknown Source) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51) at akka.http.scaladsl.util.FastFuture$$$Lambda$1319/0x000000084096d840.apply(Unknown Source) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$407/0x0000000840488440.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) "application-akka.actor.default-dispatcher-18": at com.patson.model.Airport.income$lzycompute(Airport.scala:61)
• waiting to lock <0x00000006cdffecb8> (a com.patson.model.Airport) at com.patson.model.Airport.income(Airport.scala:61) at com.patson.model.CampaignDelegateTask.(BusyDelegate.scala:34) at com.patson.model.DelegateTask$.$anonfun$campaign$1(BusyDelegate.scala:18) at com.patson.model.DelegateTask$.$anonfun$campaign$1$adapted(BusyDelegate.scala:18) at com.patson.model.DelegateTask$$$Lambda$1970/0x0000000840c13840.apply(Unknown Source) at com.patson.data.DelegateSource$.loadBusyDelegatesByCampaigns(DelegateSource.scala:213) at com.patson.data.AirportSource$.getCampaignBonuses(AirportSource.scala:86) at com.patson.data.AirportSource$.loadAirportsByQueryString(AirportSource.scala:323) at com.patson.data.AirportSource$.loadAirportsByCriteria(AirportSource.scala:42) at com.patson.data.AirportSource$.loadAirportById(AirportSource.scala:519) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:48) at com.patson.util.AirportCache$DetailedLoader.load(AirportCache.scala:46) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3451) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2200) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
• locked <0x00000006de62e508> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) at com.google.common.cache.LocalCache.get(LocalCache.java:3855) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3878) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4780) at com.patson.util.AirportCache$.getAirport(AirportCache.scala:18) at com.patson.model.AirportRating$.rateAirport(AirportRating.scala:34) at com.patson.model.Airport.rating$lzycompute(Airport.scala:91)
• locked <0x00000006d23a5158> (a com.patson.model.Airport) at com.patson.model.Airport.rating(Airport.scala:91) at com.patson.model.AirlineBase.getUpkeep$lzycompute(AirlineBase.scala:24)
• locked <0x00000006de62e548> (a com.patson.model.AirlineBase) at com.patson.model.AirlineBase.getUpkeep(AirlineBase.scala:22) at controllers.package$AirlineBaseFormat$.writes(package.scala:318) at controllers.package$AirlineBaseFormat$.writes(package.scala:297) at play.api.libs.json.LowPriorityWrites.$anonfun$traversableWrites$2(Writes.scala:334) at play.api.libs.json.LowPriorityWrites$$Lambda$1585/0x0000000840ac7040.apply(Unknown Source) at scala.collection.immutable.List.foreach(List.scala:312) at play.api.libs.json.LowPriorityWrites.$anonfun$traversableWrites$1(Writes.scala:333) at play.api.libs.json.LowPriorityWrites$$Lambda$1584/0x0000000840ac6840.apply(Unknown Source) at play.api.libs.json.Writes$$anon$6.writes(Writes.scala:157) at play.api.libs.json.Json$.toJson(Json.scala:187) at controllers.AirlineApplication.$anonfun$getAirline$1(AirlineApplication.scala:174) at controllers.AirlineApplication$$Lambda$1790/0x0000000840b6b840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.mvc.Security$AuthenticatedBuilder.$anonfun$authenticate$1(Security.scala:199) at play.api.mvc.Security$AuthenticatedBuilder$$Lambda$1612/0x0000000840ade840.apply(Unknown Source) at scala.Option.map(Option.scala:243) at play.api.mvc.Security$AuthenticatedBuilder.authenticate(Security.scala:198) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:191) at play.api.mvc.Security$AuthenticatedBuilder.invokeBlock(Security.scala:181) at play.api.mvc.ActionBuilder$$anon$10.apply(Action.scala:426) at play.api.mvc.Action.$anonfun$apply$2(Action.scala:98) at play.api.mvc.Action$$Lambda$1326/0x0000000840972040.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1331/0x0000000840974c40.apply(Unknown Source) at scala.util.Try$.apply(Try.scala:212) at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:184) at play.api.libs.streams.StrictAccumulator$$Lambda$1327/0x0000000840972840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at scala.Function1.$anonfun$andThen$1(Function1.scala:85) at scala.Function1$$Lambda$1293/0x0000000840955840.apply(Unknown Source) at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:219) at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:441) at play.core.server.AkkaHttpServer$$Lambda$1318/0x000000084096c840.apply(Unknown Source) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41) at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51) at akka.http.scaladsl.util.FastFuture$$$Lambda$1319/0x000000084096d840.apply(Unknown Source) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$407/0x0000000840488440.apply$mcV$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Found 2 deadlocks.

[patsonluk]

The current workaround https://github.com/patsonluk/airline/commit/402575db44112913412e8fd256d0b9139fe70350

A more complete fix might be either breaking the cyclic relationship (airport -> campaign in range -> principle airport) or changing how lazy loading works for Airport - perhaps introduce a new mechanism that lazy load w/o locking the whole class and only on the lazy loaded variable. This does NOT avoid all deadlocks by loading though

Lazy loading cause deadlock is a complicated issue, perhaps we will engineer such solution if we have more deadlocks due to lazy loading. For now the workaround is probably good enough