search

elixir-ecto / db_connection

Database connection behaviour
http://hexdocs.pm/db_connection/DBConnection.html
309 stars 112 forks source link

[Question] Refreshing db connections when there is a change to DNS record for the db host #98

Closed sadir closed 7 years ago

sadir commented 7 years ago

Hi,

I am working on a project which uses Ecto and MariaDB in the backend. Our MariaDB instance is provided by Amazon RDS. Recently, we have enabled multi az on our RDS instances which enables automatic failover to a secondary. When this happens, Amazon switches over the DNS record for the RDS instance to the secondary, which takes about 2 minutes.

Having said that, when that happens our application fails anyway and requires a restart because the db connections don't move over with the DNS record. We believe that the ip address is cached which causes this behaviour.

Looking around, this SO post has a comment stating we can configure TTL behaviour for dns caching through the erlang library inet. Can you please confirm that db_connection (and therefore Ecto) use inet under the hood? Will configuring inet's cache_refresh option help here?

ericmj commented 7 years ago

Can you please confirm that db_connection (and therefore Ecto) use inet under the hood?

Yes, all DNS lookups will go through inet.

Will configuring inet's cache_refresh option help here?

Don't know, I think you will have to try.

sadir commented 7 years ago

OK thanks - closing this then.

AndrewDryga commented 7 years ago

@sadir Can you write results of your research here? This is a common pattern for Kubernetes and I'd like to know how you would solve this issue :).

sadir commented 7 years ago

@AndrewDryga sure!

So digging around I noticed that inet uses native lookups as the default, not the Erlang DNS client inet_res(3).

Native lookup (system calls) is always the default resolver method. This is true for all platforms, except VxWorks and OSE Delta where 'file' or 'dns' is used (in that priority order).

Therefore inet itself couldn't be doing any dns caching because you have to use the inet_res(3) method to do that:

{cache_refresh, Time}. Time = integer() Set how often (in milliseconds) the resolver cache for inet_res(3) is refreshed (that is, expired DNS records are deleted). Defaults to 1 hour.

This made me question whether I was right about my original assessment that there was any caching whatsoever. We are using the ubuntu operating system and with linux based OSs it is very unlikely that you will be doing any DNS caching at the OS level.

I went back to do a deeper analysis and discovered that the db connections did actually fail over, however one of our applications was having other issues as a result of the failover which was impacting our stack at a wider scale. As a result there wasn't much else to do here.

Hope that helps 😄