Improve IndexFactoryBean's resilience and options for handling GemFire IndexExistsExceptions and IndexNameConflictExceptions [DATAGEODE-14]

[spring-projects-issues]

John Blum opened DATAGEODE-14 and commented

PROBLEM DESCRIPTION

Currently, there is a race condition / bug in Apache Geode where an Index creation can be in a "pending" state for a particular Geode peer member node in the cluster having a PARTITION Region on which the Index will be created.

This Apache Geode race condition is negatively impacting Spring Data Geode's (SDG) IndexFactoryBean when a peer member joining this cluster, configured with SDG, defines the same PARTITION Region and the same, associated Index.

This Apache Geode race condition affects SDG's IndexFactoryBean because the "check" for an "existing" (already defined/created) Index (by "name") occurs before the actual Index creation logic (i.e. using Geode's QueryService.createIndex(..) method calls) in SDG's IndexFactoryBean.

Essentially, Geode's QueryService.getIndexes() used by SDG's IndexFactoryBean to check for an "existing" Index returns "nothing" (!), even when the Index is in a "pending" state.

An Index will be in a "pending" state (represented as a FutureTask internally by Geode) when the Index has already been created by another peer member node in the same cluster having the same PARTITION Region (PR) and the same Index definition. It is likely that this "other' peer member node in the cluster was started first. Additionally, all PR Index definitions are "distributed" to other peer nodes in the same cluster hosting the same PR. When this happens, these other nodes will represent the "pending" Index definition as a FutureTask. And, while the QueryService.getIndexes() is not a completely blocking call (i.e. does not wait on the "pending" (FutureTask based) Indexes), the QueryService.createIndex(..) does! Therefore, it is possible that this Apache Geode induced race condition can negatively impact SDG's IndexFactoryBean as already stated.

CHANGES

So, the purpose of this JIRA "improvement" will be to make SDG's IndexFactoryBean impervious to any ill Geode behavior involving Indexes, PRs or not.

Additionally, I will maintain SDG's "fail fast" behavior with the added option to either "override" existing Indexes when Geode's IndexExistsException is thrown, or "ignore" the Index that would be created by SDG's IndexFactoryBean when Geode's IndexNameConflictException is thrown.

Finally, SDG will strive to provide the developer with more informative Exception handling, even instructing the developer on the proper course of action

---

Attachments:

• Spring Data GemFire Reference Guide - Configuring an Index.pdf (198.00 kB)

Referenced from: commits https://github.com/spring-projects/spring-data-gemfire/commit/0db66018ebcba88773fcf5565a7d5bef22ab2e90

[spring-projects-issues]

John Blum commented

Note, 1 possible workaround to this Apache Geode induced problem is to "conditionally" define the Index on the PR in each peer member node. This can be easily accomplished using Spring's bean definition profiles.

This works quite nicely especially when several Geode Servers in a cluster are all configured with the same Spring (Data Geode) configuration meta-data.

Essentially, the "Index bean definition" can be wrapped in a Spring Profile. For example...

XML

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" 
       xmlns:gfe="http://www.springframework.org/schema/gemfire" 
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/gemfire http://www.springframework.org/schema/gemfire/spring-gemfire.xsd
">

    <!-- other bean definitions here -->

    <beans profile="EnableIndex">
        <gfe:index name="SharePartitionRegionIndex" expression=".." from="/SomePartitionRegion" .../>
    </beans>

</beans>

JAVA

@Configuration
class GeodeConfiguration {

  // other @Bean definitions defining Geode objects here...

   @Bean("SharedPartitionRegionIndex")
   @Profile("EnableIndex")
   public IndexFactoryBean sharedPrIndex(GemFireCache gemfireCache) {

      IndexFactoryBean indexFactory = new IndexFactoryBean();

      indexFactory.setCache(gemfireCache);
      indexFactory.setExpression("...");
      indexFactory.setFrom("/SomePartitionRegion");
      ...

      return indexFactory;
  }
}

Then all you need do is launch the Spring (Data Geode) configured, Geode peer cache application with the JVM System Property...

-Dspring.profiles.active=EnableIndex.

This Spring Profile (i.e. "EnableIndex) should only be enabled on 1 of the peer member nodes hosting the same PR, defining the same Index. If it is defined on more that 1 peer member node having the same PR and Index definition, then there will be a possibility of a IndexNameConflictException.

[spring-projects-issues]

John Blum commented

Final changes resolving this ticket include:

1. override now defaults to false.

This is a significant behavioral change to SDG so please adjust accordingly. It is atypical for the Spring Data team to introduce a behavioral change of this magnitude in any Service Release. However, this decision was not made lightly and is directly due to the race condition / bug in Apache Geode. In order for SDG to do the right thing and function correctly, the override default had to change to false.

2. Introduced a new property/attribute, ignoreIfExists on the o.s.d.g.IndexFactoryBean or...

<gfe:index id="myIndex"  expression=".." from=".." ignore-if-exists="true"/>

This option effectively "ignores" the Index that would have been created by the Index bean definition / declaration in Spring config (specified with the <gfe:index/> element using the SDG XML namespace or the IndexFactoryBean when using Spring Java config).

3. Both ignoreIfExists and override are applicable to either the IndexExistsException or the IndexNameConflictException. Each option performs a slightly different action depending on the Exception. You should make sure you understand each option relative to the type of Index Exception thrown by Geode before using the option.

Both options may be used but ignoreIfExists takes precedence over override.

Read the updated, attached documentation from the Spring Data Geode Reference Guide on Configuring an Index for more details