bitwalker / distillery

Simplify deployments in Elixir with OTP releases!
MIT License
2.96k stars 398 forks source link

Race condition issue with generated sys.config #568

Closed imranismail closed 5 years ago

imranismail commented 5 years ago

Steps to reproduce

There are times where the app would boot up before the sys.config generation resolves.

And db_connection/ecto would complain about missing database config.

Rebooting the app resolves the issue.

Verbose Logs

Paste the output of the release command you ran with the --verbose flag below in the summary tags (this helps keep the issue easy to navigate):

``` 14:18:13.253 [error] GenServer #PID<0.2907.0> terminating ** (RuntimeError) Connect raised a KeyError error. The exception details are hidden, as they may contain sensitive data such as database credentials. (elixir) lib/keyword.ex:386: Keyword.fetch!/2 (postgrex) lib/postgrex/protocol.ex:98: Postgrex.Protocol.connect/1 (db_connection) lib/db_connection/connection.ex:135: DBConnection.Connection.connect/2 (connection) lib/connection.ex:622: Connection.enter_connect/5 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3 Last message: nil ```
imranismail commented 5 years ago

This is a deal breaker for us. We are still not confident to deploy distillery 2.0 because of this

arjan commented 5 years ago

How are you configuring Ecto? Would be handy to provide some more details on your config situation.

imranismail commented 5 years ago

I am using the Elixir config provider for configuring our Ecto repos

On Thu, 11 Oct 2018, 3:32 pm Arjan Scherpenisse, notifications@github.com wrote:

How are you configuring Ecto? Would be handy to provide some more details on your config situation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bitwalker/distillery/issues/568#issuecomment-428851160, or mute the thread https://github.com/notifications/unsubscribe-auth/AGkJPy2CJxkwFzLxuGjPlhRCSMqQORlmks5ujvPxgaJpZM4XW3MG .

aquarhead commented 5 years ago

Is the config provider reading from the network? Is it possible to output the value during running config provider phase?

imranismail commented 5 years ago

How do I do that @aquarhead

aquarhead commented 5 years ago

I'm not using config providers (yet), so just wondering whether you can use IO.inspect and check whether values exist at that point (first start). Or write out the values to another file (if stdout is not attachable)

imranismail commented 5 years ago

I will set this up and report back.

But just to clarify I am not using any sort of config providers over the network.

I'm using the built in Elixir config provider.

Generated sys.config looks about right, however the config is not loaded.

imranismail commented 5 years ago

@aquarhead @bitwalker I've confirmed this via attach command after start command.

Had to run the start then attach command combination several times to reproduce the error.

The results are

[
  {App.Repo, [adapter: Ecto.Adapters.Postgres]}
]

When it should be as per sys.config

[
  {App.Repo, [
     adapter: Ecto.Adapters.Postgres,
     username: "redacted",
     password: "redacted",
     hostname: "redacted",
     database: "redacted",
     pool_size: 10,
     pool_overflow: 20,
     port: 5432
  ]}
]
imranismail commented 5 years ago

Any update on this?

cvortmann commented 5 years ago

I've left a couple of comments on https://github.com/bitwalker/distillery/issues/275#issuecomment-459664281 about a race condition with sys.config generation that seems to be related to the issue being discussed here.

bitwalker commented 5 years ago

This should no longer be an issue with the latest versions, a signficant amount has changed since this issue was first reported