customink / activerecord-aurora-serverless-adapter

ActiveRecord Adapter for Amazon Aurora Serverless
https://technology.customink.com/blog/2020/01/03/migrate-your-rails-app-from-heroku-to-aws-lambda/
MIT License
66 stars 7 forks source link

Support Reconnects #11

Closed metaskills closed 4 years ago

metaskills commented 4 years ago

Very common to allow the Aurora Serverless autoPause after some period of time. When this happens and Rails tries to connect, you will see an error like this pretty quickly.

Aws::RDSDataService::Errors::BadRequestException (Communications link failure)
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

At one time I had code in place to handle this but removed it due to the issues from #8 for flaky tests because I assumed it was due to the Retriable (https://github.com/kamui/retriable) gem. It was not, so it is safe to add this back. Here is quick implementation I had that we can add back. I liked this method since it meant we only used retrys for the initial connect and not all the time for each execute statement.

def execute_statement_retry(sql)
  if @connected
    execute_statement(sql)
  else
    auto_paused_retry { execute_statement(sql) }
  end
end

def execute_statement(sql)
  raw_client.execute_statement({
    # ...
  }).tap do |r|
    @connected = true
    # ...
  end
end

private

def auto_paused_retry
  ekls = Aws::RDSDataService::Errors::BadRequestException
  emsg = /last packet sent successfully to the server was/
  rtry = Proc.new { Rails.logger.info 'Aurora auto paused, retrying...' }
  optn = { on: { ekls => emsg }, on_retry: rtry, tries: 10 }
  Retriable.retriable(optn) { yield }
end