B-51009: add Around Advice that knows how to retry connection failures from Postgres/Tomcat JDBC connection pool

[shintasmith]

This PR adds an Around Advice that can be used with the Atom Hopper JDBC adapter.

The main use case I would like to solve with this Advice is this:

• we are using Load Balancer (LB) VIP to connect the DB server. When we upgrade our LB software, we often times have to sever the existing connections and have the app server failover to the fail-over LB VIP. When existing connections are severed, the app server (i.e.: Atom Hopper) returns 500s to clients. With this Advice, you can configure Atom Hopper to retry up to a configurable retries and a fixed wait time between each retry. Thus, the next time we upgrade our LB software, we do not have to return 500s, thus greatly improving our availability numbers.

Another common use case that other people are more likely to experience is this:

• flaky network connection between the app server and Database server

This Advice only works when you use Atom Hopper JDBC adapter (in adapter/jdbc directory). It relies on the fact that this Atom Hopper JDBC adapter uses Spring's org.springframework.jdbc.core.JdbcTemplate.

Adding this Advice, does introduce the AspectJ jar files (3 of them: aspectrt, aspectj weaver, and cglib) into the atomhopper.war. Performance tests will be done later to verify if there are huge performance impacts with the AspectJ library. Since the use of this Advice is done from configuration, we can easily back out from using it.

To use this Advice, you will need to add something like this in the application-context.xml file:

  <bean id="jdbcRetryOnFailureAdvice" class="org.atomhopper.jdbc.adapter.JdbcRetryOnFailureAdvice">
      <property name="maxRetriesOnConnDrop" value="5"/>
      <property name="retryWaitInMillis" value="5000"/>
      <property name="sqlStatesToRetry" value="08003 08001 57P01"/>
  </bean>

  <aop:config proxy-target-class="true">
      <aop:aspect id="jdbcRetryOnUpdate" ref="jdbcRetryOnFailureAdvice">
          <aop:pointcut id="retryUpdatePointCut"
                        expression="execution(* org.springframework.jdbc.core.JdbcTemplate.update(..))"/>
          <aop:around   pointcut-ref="retryUpdatePointCut"
                        method="retryOnFailure"/>
      </aop:aspect>
      <aop:aspect id="jdbcRetryOnQuery" ref="jdbcRetryOnFailureAdvice">
          <aop:pointcut id="retryQueryPointCut"
                        expression="execution(* org.springframework.jdbc.core.JdbcTemplate.query(..))"/>
          <aop:around   pointcut-ref="retryQueryPointCut"
                        method="retryOnFailure"/>
      </aop:aspect>
  </aop:config>

The JdbcRetryOnFailureAdvice class has the following properties:

• maxRetriesOnConnDrop - how many times to retry an operation
• retryWaitInMillis - how many milliseconds to wait before performing the next retry
• sqlStatesToRetry - this is the SQL states string thrown by the underlying database driver. For any of these states listed here (white space separated), the operation will be retried

In the above examples, the Around Advice is configured to surround the JdbcTemplate.update() and JdbcTemplate.query() method calls.

[shintasmith]

I should also add that I looked into Tomcat's JDBC connection pool's testOnBorrow and testOnReturn configuration. Although, these properties address the issue of application getting "invalid" or "closed" connections, they do not always work for the use cases I mentioned above. These properties only guarantee that connections are good when they are taken out from the Connection Pool and returned. It is still possible connections are severed after application code fetches a connection from the Connection Pool, in which case you still do need to put try-catch around your JdbcTemplate calls.

I also explored creating custom Tomcat JDBC Interceptors. Those also do not work for me for the same reasons.