RPC call fails

[matrinox]

This is what I get:

05:34 node:full_restart
      01 cd /home/deploy/capistrano-app; for i in {1..10}; do bin/app ping && break || true; sleep 1; done
      01 Node 'app@127.0.0.1' not responding to pings.
      01 Node 'app@127.0.0.1' not responding to pings.
      01 pong
    ✔ 01 deploy@stage-erlang02 5.171s
      02 bin/app rpc Elixir.Carafe execute_elixir '
'\ \ \ \ \ \ \ \ fn\ -\>\ Application.started_applications\ \|\>\ Enum.any\?\(fn\ info\ -\>\ elem\(info,\ 0\)\ \=\=\ :app\ end\)\ e…
      01 pong
    ✔ 01 deploy@stage-erlang01 5.229s
      02 RPC to 'app@127.0.0.1' failed: {'EXIT',
      02                                                   {undef,
      02                                                    [{'Elixir.Carafe',
      02                                                      execute_elixir,
      02                                                      ["\n        fn -> Application.started_applications |> Enum.any?(fn info -> elem(info, 0) == :app end) end\n        |> Stream.repeatedly\n  …
      02 RPC to 'app@127.0.0.1' failed: {'EXIT',
      02                                                      []},
      02                                                   {undef,
      02                                                    [{'Elixir.Carafe',
      02                                                      execute_elixir,
      02                                                     {rpc,
      02                                                      ["\n        fn -> Application.started_applications |> Enum.any?(fn info -> elem(info, 0) == :app end) end\n        |> Stream.repeatedly\n  …
      02                                                      '-handle_call_call/6-fun-0-',
      02                                                      []},
      02                                                      5,
      02                                                     {rpc,
      02                                                      '-handle_call_call/6-fun-0-',
      02                                                      5,
      02                                                      [{file,"rpc.erl"},
      02                                                      [{file,"rpc.erl"},
      02                                                       {line,197}]}]}}
      02                                                       {line,197}]}]}}
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deploy@stage-erlang01: bin/app exit status: 1
bin/app stdout: RPC to 'app@127.0.0.1' failed: {'EXIT',
                                                  {undef,
                                                   [{'Elixir.Carafe',
                                                     execute_elixir,
                                                     ["\n        fn -> Application.started_applications |> Enum.any?(fn info -> elem(info, 0) == :app end) end\n        |> Stream.repeatedly\n        |> Stream.each(fn running -> unless running, do: :timer.sleep(250) end)\n        |> Enum.any?\n      "],
                                                     []},
                                                    {rpc,
                                                     '-handle_call_call/6-fun-0-',
                                                     5,
                                                     [{file,"rpc.erl"},
                                                      {line,197}]}]}}
bin/app stderr: Nothing written
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
SSHKit::Command::Failed: bin/app exit status: 1
bin/app stdout: RPC to 'app@127.0.0.1' failed: {'EXIT',
                                                  {undef,
                                                   [{'Elixir.Carafe',
                                                     execute_elixir,
                                                     ["\n        fn -> Application.started_applications |> Enum.any?(fn info -> elem(info, 0) == :app end) end\n        |> Stream.repeatedly\n        |> Stream.each(fn running -> unless running, do: :timer.sleep(250) end)\n        |> Enum.any?\n      "],
                                                     []},
                                                    {rpc,
                                                     '-handle_call_call/6-fun-0-',
                                                     5,
                                                     [{file,"rpc.erl"},
                                                      {line,197}]}]}}
bin/app stderr: Nothing written
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/command.rb:99:in `exit_status='
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/netssh.rb:167:in `execute_command'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:141:in `block in create_command_and_execute'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:141:in `tap'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:141:in `create_command_and_execute'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:74:in `execute'
~/.rvm/gems/ruby-2.3.1/gems/carafe-0.2.0/lib/carafe.rb:99:in `execute_elixir'
~/.rvm/gems/ruby-2.3.1/gems/carafe-0.2.0/lib/capistrano/tasks/node.rake:95:in `block (3 levels) in <top (required)>'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:85:in `within'
~/.rvm/gems/ruby-2.3.1/gems/carafe-0.2.0/lib/capistrano/tasks/node.rake:83:in `block (2 levels) in <top (required)>'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/backends/abstract.rb:29:in `run'
~/.rvm/gems/ruby-2.3.1/gems/sshkit-1.14.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy => node:full_restart
zlib(finalizer): the stream was freed prematurely.

I've replaced node:full_restart with node:stop-if-running and node:start instead. I've also included carafe in my mix dependancies but it doesn't show up in the remote console when I try to autocomplete Elixir.Carafe. Any idea?

[schnittchen]

Can you check on the build server whether there's _build/$stage/lib/carafe?

[schnittchen]

Does autocompleting for Carafe work locally?

Do you enumerate your dependent applications like before Elixir 1.4 maybe? Then you need to add carafe there.

[matrinox]

Can you check on the build server whether there's _build/$stage/lib/carafe?

ls $BUILD_PATH/_build/staging/lib/carafe/ 👍

Does autocompleting for Carafe work locally?

No... I have it in my deps but it's not loaded

Do you enumerate your dependent applications like before Elixir 1.4 maybe? Then you need to add carafe there.

I'm using Elixir 1.4.4 and OTP/20. I haven't don't that. Let me try that

[matrinox]

I was able to get it to work by just typing Carafe. and then autocompleting with tab. Then after that, Carafe is in the autocomplete. Adding it to :applications or :extra_applications didn't change that behaviour.

Ok so you don't need to add it as a dependancy and the docs are correct. But what I did doesn't work on the deployed server so same problem. I'll try to deploy with the dependancies added

[matrinox]

On a somewhat different topic, is carafe compatible with Capistrano rollbacks? I want to design this in such a way that it's similar to Capistrano deploys in ruby apps.

[schnittchen]

I have no experience with rollbacks, so I'm not going to implement that. However, PRs are always welcome.

Personally, I have been fine for years with just pushing forward ;)

[schnittchen]

Actually, I missed that you consistently wrote about Elixir.Carafe. The reason you see that in the error message is that it comes from Erlang, where Elixir names are mangled to be namespaced like that.

More precisely, Carafe is essentially syntactic sugar for :"Elixir.Carafe".

[schnittchen]

Did you manage to deploy? Or is carafe missing from the built release?

[matrinox]

I tried Carafe as well but you can do either, I'm pretty sure. I ended up just writing our own that is closer to the normal Capistrano flow. By normal I mean keeps the deploy task and just adds hooks around that flow. If I learn anything about rollbacks, I'd be happy to contribute back. Thanks!

[schnittchen]

Sorry to hear that.

[aligo]

@matrinox Same error i got, has be resolved by adding included_applications: [:carafe] to mix.exs.

[matrinox]

Thanks @aligo

[schnittchen]

This really shouldn't be necessary. Carafe should be automatically part of the release as soon as it is declared as a dep, provided elixir is able to infer applications correctly (https://sergiotapia.me/application-inference-in-elixir-1-4-ae9e43e90301?gi=7a4588b0d07e).

Things are a bit more twisted in the case of an umbrella, see https://hexdocs.pm/carafe/configurationusage.html#umbrella-project-deployments.

[matrinox]

Umbrella has been a headache ever since we've used it.

Concerning https://github.com/schnittchen/carafe/blob/master/lib/capistrano/tasks/node.rake#L89.

What is the purpose of that? Is there another way of accomplishing this? The less that is included in the production app the better and I was able to create a version that didn't use it but I also have yet to test the reliability without it

[schnittchen]

@matrinox that line is simply waiting for the OTP app to be up. In case of an umbrella app, that would be the top app in the app dependency tree.

If you omit that you might be fine sometimes, but you will run into a race condition sooner or later.

Yes it's ugly, and I am thinking of moving it over into the elixir part of carafe one day. Before I do that, this code would not be part of a release build and hence not part of a production app.

[schnittchen]

Actually I think I was wrong in my last comment, it's not race conditions that are the problem. Sometimes the application won't boot up properly (when the top level supervisor terminates), and without this polling loop we would not notice this.

[matrinox]

@schnittchen Couldn't we just ping it, with bin/#{script} ping? Or does that sometimes give false or premature positives?

[schnittchen]

In case of a boot failure, there can be a short time when the node is up and pingeable.

[matrinox]

Ping twice in a row with a 1 second sleep?

[schnittchen]

Note edeliver does something similar, albeit in a different way. https://github.com/edeliver/edeliver/blob/master/lib/edeliver/startup_progress.ex#L23

It looks like relying on ping is not a good idea. However, you are free to try other ways by "overwriting" the node:full_restart task. Let me know when that works for you for a fair number of deploys!