Cannot build a release for a simple project

[mingchuno]

First showing the error from running mix.release

Building release with MIX_ENV=dev.
{{case_clause,
     {'EXIT',
         {function_clause,
             [{filename,join,[[]],[{file,"filename.erl"},{line,393}]},
              {erl_tar,split_filename,4,[{file,"erl_tar.erl"},{line,471}]},
              {erl_tar,create_header,3,[{file,"erl_tar.erl"},{line,400}]},
              {erl_tar,add1,4,[{file,"erl_tar.erl"},{line,323}]},
              {systools_make,add_to_tar,3,
                  [{file,"systools_make.erl"},{line,1879}]},
              {lists,foreach,2,[{file,"lists.erl"},{line,1337}]},
              {systools_make,'-add_applications/5-fun-0-',6,
                  [{file,"systools_make.erl"},{line,1569}]},
              {lists,foldl,3,[{file,"lists.erl"},{line,1262}]}]}}},
 [{systools_make,'-add_applications/5-fun-0-',6,
      [{file,"systools_make.erl"},{line,1569}]},
  {lists,foldl,3,[{file,"lists.erl"},{line,1262}]},
  {systools_make,add_applications,5,[{file,"systools_make.erl"},{line,1568}]},
  {systools_make,mk_tar,6,[{file,"systools_make.erl"},{line,1562}]},
  {systools_make,mk_tar,5,[{file,"systools_make.erl"},{line,1538}]},
  {systools_make,make_tar,2,[{file,"systools_make.erl"},{line,336}]},
  {rlx_prv_archive,make_tar,3,[{file,"src/rlx_prv_archive.erl"},{line,83}]},
  {relx,run_provider,2,[{file,"src/relx.erl"},{line,308}]}]}
==> ERROR: "Failed to build release. Please fix any errors and try again."

my mix.exs is like this. This is a simple application that connect both rabbitmq server and mqtt server. I have try to add/remove applications from the applications list but still fail. Running on: Ubuntu 14.04 Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false] IEx 1.2.0 Mix 1.2.0

defmodule Websocket.Mixfile do
  use Mix.Project

  def project do
    [app: :websocket,
     version: "0.0.1",
     elixir: "~> 1.2",
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     deps: deps]
  end

  # Configuration for the OTP application
  #
  # Type "mix help compile.app" for more information
  def application do
    [
      applications: [
        :logger, 
        :amqp, 
        :emqttc, 
        :exprotobuf, 
        :poison
      ],
      mod: {Websocket.Application, []}
    ]
  end

  # Dependencies can be Hex packages:
  #
  #   {:mydep, "~> 0.3.0"}
  #
  # Or git/path repositories:
  #
  #   {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"}
  #
  # Type "mix help deps" for more examples and options
  defp deps do
    [
      {:amqp, "~> 0.1.4"},
      {:exprotobuf, "~> 1.0.0-rc1"},
      {:poison, "~> 2.1"},
      {:emqttc, git: "https://github.com/emqtt/emqttc.git"},
      {:exrm, "~> 1.0.0-rc8"},
      {:getopt, "~> 0.8.2", [optional: false, hex: :getopt, manager: :rebar, override: true]},
      {:cf, "~> 0.2.1", override: true}, # remove this later
      {:erlware_commons, github: "erlware/erlware_commons", override: true} # remove this later
    ]
  end
end

This is what I get when ls from project root

> ~/dev/websocket$ ls rel/files/
boot  boot.bat  boot_shim  boot_shim.bat  relx.config  sys.config
> ~/dev/websocket$ ls rel/websocket/
bin  erts-7.2  lib  releases
> ~/dev/websocket$ ls rel/websocket/releases/
0.0.1  RELEASES  start_erl.data
> ~/dev/websocket$ ls rel/websocket/releases/0.0.1/
start_clean.boot  sys.config  vm.args  websocket.bat  websocket.boot  websocket.rel  websocket.script  websocket.sh
> ~/dev/websocket$ 

Any insight?

[mingchuno]

running MIX_ENV=prod mix compile and then MIX_ENV=prod mix release --dev work. I can even run my application. Looking from the source code, I think maybe it fail when packing the tarball?

log by running MIX_ENV=prod mix release --verbosity=verbose

https://gist.github.com/mingchuno/d11aa9527a3bcf9bdf45

[mingchuno]

damn, nail down to the same issue in

Problems with building release in project including exprotobuf · Issue #33 · bitwalker/exprotobuf https://github.com/bitwalker/exprotobuf/issues/33

I am using porotobuf too....

[bitwalker]

I'll dig into this this weekend!

[mingchuno]

@bitwalker Thanks a lot. Anything I can help?

[bitwalker]

Could you gist the output of tree ./rel? If you don't have tree installed, I highly recommend it, but regardless of the method, knowing what files are present in the release directory may help isolate the cause.

[mingchuno]

Here is my src to generate the protobuf message

defmodule Websocket.Messages do
  use Protobuf, from: Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__)), use_package_names: true
end

running $ git clean -Xdf && mix deps.get && mix deps.compile && mix compile && mix release --dev && tree ./rel > ~/Desktop/tree_rel.log got me

https://gist.github.com/mingchuno/ae2118596c167bdf0b70

[mingchuno]

I have some new discovery. the protobuf file I am using is a git submodule and contains a lot of *.proto file for developers use in the same organization. In my last example, I include them all and fail to mix release. Since I am building a toy project right now, I only need a part of the proto files as a starting point. When I change my message.ex to the below, I can build the release.

defmodule Websocket.Messages do
  use Protobuf, from:  ["./protobuf/quote/Quote.proto", "./protobuf/user/Classification.proto"], use_package_names: true
end

Will it be the chance that there are too many compiled module files, and then those files are pass though tar as args and exist the args length limit of tar or any thing else?

[sashaafm]

I think I'm having the same problem. I can only generate the release like @mingchuno said, using --dev:

sashaafm@LXLE:~/Documents/Portal/s21sec_portal$ MIX_ENV=prod mix release
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
Building release with MIX_ENV=prod.
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
==> ERROR: "Failed to build release. Please fix any errors and try again."

I'm not sure what other info I can post to help, but tell me if you need any. I'm using version 1.0.0-rc8

LXLE (Ubuntu 14.04 I think) Elixir 1.2.0 Erlang/OTP 18 erts-7.2

[mingchuno]

@sashaafm, try this git clean -Xdf && mix deps.get && mix deps.compile && mix compile && mix release --verbosity=verbose

I work around this by reducing number of protobuf to build

[bitwalker]

@mingchuno Have you checked to see what the output of Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__)) is? I really really doubt that it's the number of protobufs that matters, there is almost certainly something else going on.

[mingchuno]

@bitwalker

iex(1)> Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__))
[]
iex(2)> Path.wildcard(Path.expand("./protobuf/**/*.proto", __DIR__))
["/Users/mcor/dev/websocket/protobuf/Request.proto",
 "/Users/mcor/dev/websocket/protobuf/RequestType.proto",
 "/Users/mcor/dev/websocket/protobuf/Respond.proto",
 "/Users/mcor/dev/websocket/protobuf/Status.proto",
 "/Users/mcor/dev/websocket/protobuf/chart/CandleFormat.proto",
 "/Users/mcor/dev/websocket/protobuf/chart/CandleTimeScale.proto",
 "/Users/mcor/dev/websocket/protobuf/chart/Candlestick.proto",
 "/Users/mcor/dev/websocket/protobuf/chart/Chart.proto",
 "/Users/mcor/dev/websocket/protobuf/dashboard/UniqueUserEntry.proto",
 "/Users/mcor/dev/websocket/protobuf/dashboard/WeekEntry.proto",
 "/Users/mcor/dev/websocket/protobuf/echo/Echo.proto",
 "/Users/mcor/dev/websocket/protobuf/echo/MT4HealthCheck.proto",
 "/Users/mcor/dev/websocket/protobuf/echo/SymbolTimestamp.proto",
 "/Users/mcor/dev/websocket/protobuf/event/EngineEvent.proto",
 "/Users/mcor/dev/websocket/protobuf/event/ErrorLogMessage.proto",
 "/Users/mcor/dev/websocket/protobuf/portfolio/Basic.proto",
 "/Users/mcor/dev/websocket/protobuf/portfolio/Detail.proto",
 "/Users/mcor/dev/websocket/protobuf/prompt/AccountPrompt.proto",
 "/Users/mcor/dev/websocket/protobuf/quote/Quote.proto",
 "/Users/mcor/dev/websocket/protobuf/risk/GetAllLmaxPending.proto",
 "/Users/mcor/dev/websocket/protobuf/risk/MarginCall.proto",
 "/Users/mcor/dev/websocket/protobuf/risk/NetExposure.proto",
 "/Users/mcor/dev/websocket/protobuf/risk/RiskReport.proto",
 "/Users/mcor/dev/websocket/protobuf/risk/StopOut.proto",
 "/Users/mcor/dev/websocket/protobuf/semicopy/FollowedTrade.proto",
 "/Users/mcor/dev/websocket/protobuf/semicopy/ListStarTrade.proto",
 "/Users/mcor/dev/websocket/protobuf/semicopy/PublishStarTrade.proto",
 "/Users/mcor/dev/websocket/protobuf/semicopy/StarTrade.proto",
 "/Users/mcor/dev/websocket/protobuf/stat/Request.proto",
 "/Users/mcor/dev/websocket/protobuf/stat/Respond.proto",
 "/Users/mcor/dev/websocket/protobuf/stat/Statistic.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/ClosedBy.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/OpenedBy.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/OrderBasic.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/OrderType.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/Request.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/Respond.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/Ticket.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/TicketStatus.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/TicketType.proto",
 "/Users/mcor/dev/websocket/protobuf/trade/TradeAction.proto",
 "/Users/mcor/dev/websocket/protobuf/tradelog/GetTradeLog.proto",
 "/Users/mcor/dev/websocket/protobuf/tradelog/TradeLog.proto",
 "/Users/mcor/dev/websocket/protobuf/user/Account.proto",
 "/Users/mcor/dev/websocket/protobuf/user/AccountPasswordPair.proto",
 "/Users/mcor/dev/websocket/protobuf/user/CRMBalance.proto",
 "/Users/mcor/dev/websocket/protobuf/user/CRMSearchBalance.proto",
 "/Users/mcor/dev/websocket/protobuf/user/Classification.proto",
 "/Users/mcor/dev/websocket/protobuf/user/CreateAccount.proto",
 "/Users/mcor/dev/websocket/protobuf/user/Deposit.proto", ...]
iex(3)>

if I use Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__)) in my code, i can mix compile and iex -S mix to run my application happily. Even mix release --dev work and runable.

[sashaafm]

@mingchuno I used the command you provided but it seems to have produced a dev release instead of a prod one? I need a production release :(

[mingchuno]

@sashaafm try prepend MIX_ENV=prod before the command

[bitwalker]

@mingchuno The issue appears to me to be that when building a release with ../protobuf, during execution of the overlays for the release,filename:join([]) is called, which is invalid because it requires at least one filename in that list, this results in a FunctionClauseError being thrown. Are you using a custom relx.config file by any chance?

[mingchuno]

@bitwalker for my project, I use standard mix.exs and standard mix config. But I am not sure if any transitive deps using custom relx.conf. I will take a look tmr.

Btw, this work for me now

use Protobuf, from:  [
    "./protobuf/quote/Quote.proto",
    "./protobuf/user/Classification.proto",
    "./protobuf/event/EngineEvent.proto",
    "./protobuf/trade/Ticket.proto",
    # skipping some more....
    "./protobuf/trade/OpenedBy.proto"
  ], use_package_names: true

[jeromedoyle]

I am having the exact same issue on a phoenix 1.1.4 app. Output of tree rel/. is at https://gist.github.com/jeromedoyle/86c079feb89c5ff8b8f7. There was a commit to relx a week ago that calls filename:join and may have something to do with it. https://github.com/erlware/relx/commit/3ba0fafdbbfd5d9c3af7230b68cf88d15c5376cd

[jeromedoyle]

Tried relx 3.16. Same issue.

[jeromedoyle]

I am not using exprotobuf, but I am using {:cf, "~> 0.2.1", override: true} and {:erlware_commons, github: "erlware/erlware_commons", override: true}. Here's my mix.exs

defmodule Pipe.Mixfile do
  use Mix.Project

  def project do
    [app: :pipe,
     version: "0.0.1",
     elixir: "~> 1.0",
     elixirc_paths: elixirc_paths(Mix.env),
     compilers: [:phoenix, :gettext] ++ Mix.compilers,
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     aliases: aliases,
     deps: deps]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [mod: {Pipe, []},
     applications: [:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
                    :phoenix_ecto, :postgrex, :httpotion, :elixir_talk, :connection]]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "web", "test/support"]
  defp elixirc_paths(_),     do: ["lib", "web"]

  # Specifies your project dependencies.
  #
  # Type `mix help deps` for examples and options.
  defp deps do
    [
      {:phoenix, "~> 1.1.4"},
      {:postgrex, ">= 0.0.0"},
      {:phoenix_ecto, "~> 2.0"},
      {:phoenix_html, "~> 2.4"},
      {:phoenix_live_reload, "~> 1.0", only: :dev},
      {:gettext, "~> 0.9"},
      {:cowboy, "~> 1.0"},
      #  custom
      {:ibrowse, github: "cmullaparthi/ibrowse", tag: "v4.1.2"},
      {:httpotion, github: "jeromedoyle/httpotion", branch: "digest_auth"},
      {:elixir_talk, github: "jeromedoyle/elixir_talk"},
      {:exrm, "~> 1.0.0-rc8"},
      {:poison, "~> 2.0", override: true},
      # REMOVE LATER - fixes https://github.com/bitwalker/exrm/issues/294
      {:erlware_commons, github: "erlware/erlware_commons", override: true},
      {:cf, "~> 0.2.1", override: true}
    ]
  end

  # Aliases are shortcut or tasks specific to the current project.
  # For example, to create, migrate and run the seeds file at once:
  #
  #     $ mix ecto.setup
  #
  # See the documentation for `Mix` for more info on aliases.
  defp aliases do
    ["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
     "ecto.reset": ["ecto.drop", "ecto.setup"]]
  end
end

[jeromedoyle]

This seems to be an issue with long filenames or filepaths. In my case, I have several long module names to represent nested structs that I am decoding json into using Poison.
The module names are in the format of My.Really.Really.Long.Struct.Name.I have narrowed it down to a particular struct. When I change the name to anything less than 88 characters, building the release completes successfully. 88 characters or more, and building the tar during release fails.

[bitwalker]

@mingchuno @jeromedoyle I'm unable to troubleshoot this effectively - I can't reproduce it locally, but I'm also not sure how to. Could one of you (or anyone else watching) produce a small project which I can use to reproduce and fix this (if it's something I'm able to fix).

[jeromedoyle]

I'll see if I can make one.

[jeromedoyle]

Here ya go. https://github.com/jeromedoyle/exrm_test The release builds as is. Change the commented out line in lib/exrm_test.ex to be the long one and it fails building the release.

[jeromedoyle]

There's info about max lengths here and erl_tar mentions passing in --format=ustar. I don't know enough about elixir/erlang yet, otherwise, I would create a fix and pull request. I've renamed my module to something shorter to avoid this.

[bitwalker]

@jeromedoyle The problem is that we actually don't have to change anything in relx/erl_tar to support this - it's handled by erl_tar itself. The real problem is that systools uses erl_tar to pack/unpack releases - if the filename limit is exceeded, there just isn't anything we can do. I'm not sure what the fix for this should be, though we know the workaround of course - but ultimately I think this has to be addressed by the OTP team, if it even can be due to legacy support.

[jeromedoyle]

Maybe put something in the common issues section of the docs?

[bitwalker]

Will do. Hopefully a better solution can be found :(

[noizu]

would it be appropriate to consider another packaging mechanism if the file cap enforced by erl_tar is significantly less than the underlying operating system.

What does erlang/elixir code loading look like. do beam files need to follow a specific dir structure or can the file names be hashed to avoid exceeding caps there. For example I have some ridiculously long file names due to how protocol names are generated. (protocol + module)

[bitwalker]

We can't avoid erl_tar because release_handler/systools use it internally for unpacking releases and applying upgrades. I suppose we could maybe write a new implementation of the unpacking part and have release_handler take over from there, but I'll have to look into whether that's feasible, if it is, it will likely have to be part of my work on implementing releases in Mix. I have a feeling though that this may be a limitation driven by the fact that Erlang simply does not have long module names, not that it can't, but convention avoids namespacing, and I don't think it was something the OTP team had in mind when they designed it. I'll follow up as my work progesses with releases in Mix.

[martin-langhoff]

Hitting this with distillery as well :-/