Open jasonwang3 opened 1 year ago
Hi @jasonwang3 , can you please provide more information. What does your cache configuration look like? What objects are you storing? Do you have a resilience strategy configured? There's not much we can go forward with given just a stack trace with no context.
Hi @jhouserizer , ehcache version is 3.9.9 `<config xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://www.ehcache.org/v3' xsi:schemaLocation=" http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.0.xsd">
<persistence directory="/tmp/${ROLE}"/>
<cache alias="attachements">
<resources>
<heap unit="entries">10000</heap>
<disk unit="GB" persistent="true">4</disk>
</resources>
</cache>
<cache alias="broadcastForConnector">
<resources>
<heap unit="MB">200</heap>
<disk unit="GB" persistent="true">1</disk>
</resources>
</cache>
`
resilience strategy is defalut and object is Java normal Object. In this situation system memroy free is very low about ~170M but JVM not occurs OOM issue. I'm very confused that not use pinned method but why shown 'Storage Engine and Eviction Failed - Everything Pinned (2246 mappings)'?
This is spring applicationContext
`
</bean>
<cache:advice id="cacheAdvice" cache-manager="cacheManager">
<cache:caching cache="attachements">
<cache:cacheable method="getNotificationAttachment" key="#p0"/>
</cache:caching>
<cache:caching cache="broadcastForConnector">
<cache:cacheable method="getBroadcastForConnector" key="#p0"/>
<cache:cache-evict method="getBroadcastForConnector" key="#p0" condition="#result.broadcastSettings.isEnableIncidentZone() and #result.endDate.before(new java.util.Date())" before-invocation="false"/>
</cache:caching>
</cache:advice>`
SFAIK, pinning would kick in if you're using write-behind. I don't see that in your shared config (but there are other ways to enable it) -- are you using write-behind? if so, then my guess is that the writer is not keeping up.
So when I occurs this situation how should be handle this issue or avoid this issue? Or some listeners can handle like put failure then clear cache store?
Well, can you confirm one way or another that you are / are not using a writer (e..g write-behind)
I confirmed spring cache default is write-through
I also found when occurs this issue, ehcache put will become very slow
So this means your writer - writing to the db, isn't not executing fast enough, e.g. it takes longer to write to the db than the interval between your additions/updates to cache entries.
No sure what you can do about that - faster db, less workload, etc.
What did you mean db? Is disk or memroy? How can I avoide this issue? Slow down write elements into ehcache throughput? Or can modify some parameters?
"org.ehcache.spi.resilience.StoreAccessException: The element with key '64cbf77821dc7973d6b3c8cf' is too large to be stored in this offheap store. at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1038) at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.put(AbstractOffHeapStore.java:250) at org.ehcache.impl.internal.store.tiering.TieredStore.put(TieredStore.java:108) at org.ehcache.core.Ehcache.doPut(Ehcache.java:94) at org.ehcache.core.EhcacheBase.put(EhcacheBase.java:189) at org.ehcache.jsr107.Eh107Cache.put(Eh107Cache.java:175) at org.springframework.cache.jcache.JCacheCache.put(JCacheCache.java:95) at org.springframework.cache.interceptor.AbstractCacheInvoker.doPut(AbstractCacheInvoker.java:87) at org.springframework.cache.interceptor.CacheAspectSupport$CachePutRequest.apply(CacheAspectSupport.java:837) at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:430) at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345) at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220) at com.sun.proxy.$Proxy68.getBroadcastForConnector(Unknown Source) at com.everbridge.notification.connector.utils.ConnectorMetricsUtils.lambda$logDequeuedDelay$0(ConnectorMetricsUtils.java:156) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.everbridge.notification.connector.utils.ConnectorMetricsUtils.logDequeuedDelay(ConnectorMetricsUtils.java:151) at com.everbridge.notification.connector.ConnectorConsumerActor.getAttempts(ConnectorConsumerActor.java:62) at com.everbridge.notification.connector.ConnectorConsumerActor.lambda$createReceive$0(ConnectorConsumerActor.java:47) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) at akka.actor.ActorCell.invoke(ActorCell.scala:548) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.ehcache.shadow.org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Everything Pinned (2246 mappings) Storage Engine : org.ehcache.shadow.org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine@5dbb651b at org.ehcache.shadow.org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:86) at org.ehcache.shadow.org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:685) at org.ehcache.shadow.org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1958) at org.ehcache.shadow.org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:520) at org.ehcache.shadow.org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:739) at org.ehcache.impl.internal.store.disk.EhcachePersistentConcurrentOffHeapClockCache.compute(EhcachePersistentConcurrentOffHeapClockCache.java:158) at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1036) ... 37 common frames omitted