apache / ignite

Apache Ignite
https://ignite.apache.org/
Apache License 2.0
4.82k stars 1.9k forks source link

Question: Querying Ignite Cache with Eviction and external storage enabled #11433

Closed ratadepally closed 4 months ago

ratadepally commented 4 months ago

Hi,

We are currently working on a scenario where we have cache with some of the below configuration.

  1. Dedicated data region with min and max provided and eviction enabled with with Random-2-LRU.
  2. External storage persistence for data recovery.
  3. two backups

With the above setup we have come across a behavior where if the data from the cache reaches its limit the data is automatically getting evicted which is expected. But the issue we are currently facing is that we are not able to query the data which is evicted using SQL API as the data is not longer present in OFF HEAP memory.

Also when eviction happens can we still retrieve the data from the configured backups or they will be evicted as well?

In this case the only way to get the data is to explicitly call either of below methods.

  1. cache.get(ID) - This will load specific Key data onto off heap memory which will then be available for querying
  2. cache.loadCache() - In this case do we need to call the method every time we need to retrieve a specific key from cache data or for every transaction as the data we are looking for might not be in cache due to eviction in place?

Can you please suggest any approach where we can perform SQL queries to retrieve data with Cache eviction in place and don't have to load the data for every transaction.

Attaching sample configuration for dataregion and cache config image

image

ptupitsyn commented 4 months ago

Ignite SQL does not load data from external storage.

I suggest using native Ignite persistence instead of external storage for data recovery. In this case SQL is fully supported.

ratadepally commented 4 months ago

@ptupitsyn - thank you for the quick response.