Closed mingchuno closed 6 years ago
running MIX_ENV=prod mix compile
and then MIX_ENV=prod mix release --dev
work. I can even run my application. Looking from the source code, I think maybe it fail when packing the tarball?
log by running MIX_ENV=prod mix release --verbosity=verbose
damn, nail down to the same issue in
Problems with building release in project including exprotobuf · Issue #33 · bitwalker/exprotobuf https://github.com/bitwalker/exprotobuf/issues/33
I am using porotobuf too....
I'll dig into this this weekend!
@bitwalker Thanks a lot. Anything I can help?
Could you gist the output of tree ./rel
? If you don't have tree installed, I highly recommend it, but regardless of the method, knowing what files are present in the release directory may help isolate the cause.
Here is my src to generate the protobuf message
defmodule Websocket.Messages do
use Protobuf, from: Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__)), use_package_names: true
end
running $ git clean -Xdf && mix deps.get && mix deps.compile && mix compile && mix release --dev && tree ./rel > ~/Desktop/tree_rel.log
got me
I have some new discovery. the protobuf file I am using is a git submodule and contains a lot of *.proto
file for developers use in the same organization. In my last example, I include them all and fail to mix release
. Since I am building a toy project right now, I only need a part of the proto files as a starting point. When I change my message.ex
to the below, I can build the release.
defmodule Websocket.Messages do
use Protobuf, from: ["./protobuf/quote/Quote.proto", "./protobuf/user/Classification.proto"], use_package_names: true
end
Will it be the chance that there are too many compiled module files, and then those files are pass though tar
as args and exist the args length limit of tar
or any thing else?
I think I'm having the same problem. I can only generate the release like @mingchuno said, using --dev:
sashaafm@LXLE:~/Documents/Portal/s21sec_portal$ MIX_ENV=prod mix release
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
Building release with MIX_ENV=prod.
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
==> erlware_commons
Error evaluating rebar config script ./rebar.config.script:16: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependency defined in the script won't be available unless you add them to your Mix project
==> ERROR: "Failed to build release. Please fix any errors and try again."
I'm not sure what other info I can post to help, but tell me if you need any. I'm using version 1.0.0-rc8
LXLE (Ubuntu 14.04 I think) Elixir 1.2.0 Erlang/OTP 18 erts-7.2
@sashaafm, try this
git clean -Xdf && mix deps.get && mix deps.compile && mix compile && mix release --verbosity=verbose
I work around this by reducing number of protobuf to build
@mingchuno Have you checked to see what the output of Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__))
is? I really really doubt that it's the number of protobufs that matters, there is almost certainly something else going on.
@bitwalker
iex(1)> Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__))
[]
iex(2)> Path.wildcard(Path.expand("./protobuf/**/*.proto", __DIR__))
["/Users/mcor/dev/websocket/protobuf/Request.proto",
"/Users/mcor/dev/websocket/protobuf/RequestType.proto",
"/Users/mcor/dev/websocket/protobuf/Respond.proto",
"/Users/mcor/dev/websocket/protobuf/Status.proto",
"/Users/mcor/dev/websocket/protobuf/chart/CandleFormat.proto",
"/Users/mcor/dev/websocket/protobuf/chart/CandleTimeScale.proto",
"/Users/mcor/dev/websocket/protobuf/chart/Candlestick.proto",
"/Users/mcor/dev/websocket/protobuf/chart/Chart.proto",
"/Users/mcor/dev/websocket/protobuf/dashboard/UniqueUserEntry.proto",
"/Users/mcor/dev/websocket/protobuf/dashboard/WeekEntry.proto",
"/Users/mcor/dev/websocket/protobuf/echo/Echo.proto",
"/Users/mcor/dev/websocket/protobuf/echo/MT4HealthCheck.proto",
"/Users/mcor/dev/websocket/protobuf/echo/SymbolTimestamp.proto",
"/Users/mcor/dev/websocket/protobuf/event/EngineEvent.proto",
"/Users/mcor/dev/websocket/protobuf/event/ErrorLogMessage.proto",
"/Users/mcor/dev/websocket/protobuf/portfolio/Basic.proto",
"/Users/mcor/dev/websocket/protobuf/portfolio/Detail.proto",
"/Users/mcor/dev/websocket/protobuf/prompt/AccountPrompt.proto",
"/Users/mcor/dev/websocket/protobuf/quote/Quote.proto",
"/Users/mcor/dev/websocket/protobuf/risk/GetAllLmaxPending.proto",
"/Users/mcor/dev/websocket/protobuf/risk/MarginCall.proto",
"/Users/mcor/dev/websocket/protobuf/risk/NetExposure.proto",
"/Users/mcor/dev/websocket/protobuf/risk/RiskReport.proto",
"/Users/mcor/dev/websocket/protobuf/risk/StopOut.proto",
"/Users/mcor/dev/websocket/protobuf/semicopy/FollowedTrade.proto",
"/Users/mcor/dev/websocket/protobuf/semicopy/ListStarTrade.proto",
"/Users/mcor/dev/websocket/protobuf/semicopy/PublishStarTrade.proto",
"/Users/mcor/dev/websocket/protobuf/semicopy/StarTrade.proto",
"/Users/mcor/dev/websocket/protobuf/stat/Request.proto",
"/Users/mcor/dev/websocket/protobuf/stat/Respond.proto",
"/Users/mcor/dev/websocket/protobuf/stat/Statistic.proto",
"/Users/mcor/dev/websocket/protobuf/trade/ClosedBy.proto",
"/Users/mcor/dev/websocket/protobuf/trade/OpenedBy.proto",
"/Users/mcor/dev/websocket/protobuf/trade/OrderBasic.proto",
"/Users/mcor/dev/websocket/protobuf/trade/OrderType.proto",
"/Users/mcor/dev/websocket/protobuf/trade/Request.proto",
"/Users/mcor/dev/websocket/protobuf/trade/Respond.proto",
"/Users/mcor/dev/websocket/protobuf/trade/Ticket.proto",
"/Users/mcor/dev/websocket/protobuf/trade/TicketStatus.proto",
"/Users/mcor/dev/websocket/protobuf/trade/TicketType.proto",
"/Users/mcor/dev/websocket/protobuf/trade/TradeAction.proto",
"/Users/mcor/dev/websocket/protobuf/tradelog/GetTradeLog.proto",
"/Users/mcor/dev/websocket/protobuf/tradelog/TradeLog.proto",
"/Users/mcor/dev/websocket/protobuf/user/Account.proto",
"/Users/mcor/dev/websocket/protobuf/user/AccountPasswordPair.proto",
"/Users/mcor/dev/websocket/protobuf/user/CRMBalance.proto",
"/Users/mcor/dev/websocket/protobuf/user/CRMSearchBalance.proto",
"/Users/mcor/dev/websocket/protobuf/user/Classification.proto",
"/Users/mcor/dev/websocket/protobuf/user/CreateAccount.proto",
"/Users/mcor/dev/websocket/protobuf/user/Deposit.proto", ...]
iex(3)>
if I use Path.wildcard(Path.expand("../protobuf/**/*.proto", __DIR__))
in my code, i can mix compile
and iex -S mix
to run my application happily. Even mix release --dev
work and runable.
@mingchuno I used the command you provided but it seems to have produced a dev release instead of a prod one? I need a production release :(
@sashaafm try prepend MIX_ENV=prod
before the command
@mingchuno The issue appears to me to be that when building a release with ../protobuf
, during execution of the overlays for the release,filename:join([])
is called, which is invalid because it requires at least one filename in that list, this results in a FunctionClauseError being thrown. Are you using a custom relx.config
file by any chance?
@bitwalker for my project, I use standard mix.exs
and standard mix config. But I am not sure if any transitive deps using custom relx.conf
. I will take a look tmr.
Btw, this work for me now
use Protobuf, from: [
"./protobuf/quote/Quote.proto",
"./protobuf/user/Classification.proto",
"./protobuf/event/EngineEvent.proto",
"./protobuf/trade/Ticket.proto",
# skipping some more....
"./protobuf/trade/OpenedBy.proto"
], use_package_names: true
I am having the exact same issue on a phoenix 1.1.4 app. Output of tree rel/.
is at https://gist.github.com/jeromedoyle/86c079feb89c5ff8b8f7. There was a commit to relx a week ago that calls filename:join and may have something to do with it. https://github.com/erlware/relx/commit/3ba0fafdbbfd5d9c3af7230b68cf88d15c5376cd
Tried relx 3.16. Same issue.
I am not using exprotobuf, but I am using {:cf, "~> 0.2.1", override: true}
and {:erlware_commons, github: "erlware/erlware_commons", override: true}
. Here's my mix.exs
defmodule Pipe.Mixfile do
use Mix.Project
def project do
[app: :pipe,
version: "0.0.1",
elixir: "~> 1.0",
elixirc_paths: elixirc_paths(Mix.env),
compilers: [:phoenix, :gettext] ++ Mix.compilers,
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
aliases: aliases,
deps: deps]
end
# Configuration for the OTP application.
#
# Type `mix help compile.app` for more information.
def application do
[mod: {Pipe, []},
applications: [:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
:phoenix_ecto, :postgrex, :httpotion, :elixir_talk, :connection]]
end
# Specifies which paths to compile per environment.
defp elixirc_paths(:test), do: ["lib", "web", "test/support"]
defp elixirc_paths(_), do: ["lib", "web"]
# Specifies your project dependencies.
#
# Type `mix help deps` for examples and options.
defp deps do
[
{:phoenix, "~> 1.1.4"},
{:postgrex, ">= 0.0.0"},
{:phoenix_ecto, "~> 2.0"},
{:phoenix_html, "~> 2.4"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.9"},
{:cowboy, "~> 1.0"},
# custom
{:ibrowse, github: "cmullaparthi/ibrowse", tag: "v4.1.2"},
{:httpotion, github: "jeromedoyle/httpotion", branch: "digest_auth"},
{:elixir_talk, github: "jeromedoyle/elixir_talk"},
{:exrm, "~> 1.0.0-rc8"},
{:poison, "~> 2.0", override: true},
# REMOVE LATER - fixes https://github.com/bitwalker/exrm/issues/294
{:erlware_commons, github: "erlware/erlware_commons", override: true},
{:cf, "~> 0.2.1", override: true}
]
end
# Aliases are shortcut or tasks specific to the current project.
# For example, to create, migrate and run the seeds file at once:
#
# $ mix ecto.setup
#
# See the documentation for `Mix` for more info on aliases.
defp aliases do
["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
"ecto.reset": ["ecto.drop", "ecto.setup"]]
end
end
This seems to be an issue with long filenames or filepaths. In my case, I have several long module names to represent nested structs that I am decoding json into using Poison.
The module names are in the format of My.Really.Really.Long.Struct.Name
.I have narrowed it down to a particular struct. When I change the name to anything less than 88 characters, building the release completes successfully. 88 characters or more, and building the tar during release fails.
@mingchuno @jeromedoyle I'm unable to troubleshoot this effectively - I can't reproduce it locally, but I'm also not sure how to. Could one of you (or anyone else watching) produce a small project which I can use to reproduce and fix this (if it's something I'm able to fix).
I'll see if I can make one.
Here ya go.
https://github.com/jeromedoyle/exrm_test
The release builds as is. Change the commented out line in lib/exrm_test.ex
to be the long one and it fails building the release.
@jeromedoyle The problem is that we actually don't have to change anything in relx/erl_tar to support this - it's handled by erl_tar itself. The real problem is that systools uses erl_tar to pack/unpack releases - if the filename limit is exceeded, there just isn't anything we can do. I'm not sure what the fix for this should be, though we know the workaround of course - but ultimately I think this has to be addressed by the OTP team, if it even can be due to legacy support.
Maybe put something in the common issues section of the docs?
Will do. Hopefully a better solution can be found :(
would it be appropriate to consider another packaging mechanism if the file cap enforced by erl_tar is significantly less than the underlying operating system.
What does erlang/elixir code loading look like. do beam files need to follow a specific dir structure or can the file names be hashed to avoid exceeding caps there. For example I have some ridiculously long file names due to how protocol names are generated. (protocol + module)
We can't avoid erl_tar because release_handler/systools use it internally for unpacking releases and applying upgrades. I suppose we could maybe write a new implementation of the unpacking part and have release_handler take over from there, but I'll have to look into whether that's feasible, if it is, it will likely have to be part of my work on implementing releases in Mix. I have a feeling though that this may be a limitation driven by the fact that Erlang simply does not have long module names, not that it can't, but convention avoids namespacing, and I don't think it was something the OTP team had in mind when they designed it. I'll follow up as my work progesses with releases in Mix.
Hitting this with distillery as well :-/
First showing the error from running
mix.release
my
mix.exs
is like this. This is a simple application that connect both rabbitmq server and mqtt server. I have try to add/remove applications from the applications list but still fail. Running on: Ubuntu 14.04 Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false] IEx 1.2.0 Mix 1.2.0This is what I get when
ls
from project rootAny insight?