Tar --output-dir error

[MarkoMin]

https://github.com/erlware/relx/blob/9411219b1b4160c13450ad598276ab2f4141f0b1/priv/templates/install_upgrade_escript#L168-L171

Lets say that we have a ROOTdirectory which is a root of our release. Then we have REL_ROOT directory which is ROOT/_build/default/rel/<release_name>. Why the lines above are searching for a release tarball in REL_ROOT/releases, and not in REL_ROOT itself? I tried to fix this using --output-dir but it is impossible to specify REL_ROOT/releases as an output dir...

e.g. When i type rebar3 tar ... --output-dir _build/default/rel/<release_name>/releases it puts a tarball in _build/default/rel/<release_name>/releases/<release_name> and again, the install&upgrade script can't find it.

Am I missing something or --output-dir isn't working the way it should? Also, wouldn't REL_ROOT/releases be a better option for a default tar output directory?

[ferd]

The paths searched by the install_upgrade_escript are mostly aligned with the requirements set by release_handler:unpack_release(Name), which are mostly an OTP standard outside of our control. It expects to find files in a given location to unpack and run everything fine.

Are you running a build in prod directly with the rebar3 tree structure in it? While we could just dump that tarball at _build/<profile>/rel/<release_name>/releases/ and let it do its thing, there is actually no intended relationship between the rebar3 _build directory structure and layout and what you would use in production.

The intended use is that you pick whatever tarball rebar3 builds and use that as a build artifact entirely distinct from what rebar3 outputs in _build for your deployments. None of _build should be kept otherwise. So we just output the tarball to a place that is unlikely to clash with the _build/<profile>/rel/<release_name> artifact in subsequent operations (such that generating the tarball many times, or doing relups multiple times should not contain a previous tarball by accident) and then it's up to you to put this wherever you want in production.

[MarkoMin]

Thanks for the explanation, but I just want to point out that even when I specify the --output-dir the tarball doesn't end up in specified directory but somewhere else.

I understand the thing about the defaults, but shouldn't user have the flexibility to put the tarball wherever he wants?

[MarkoMin]

I hit this problem once again - but now I your answer better, ignore my previous answer.

Everything you said is true for production and I got no problems there - but I encountered this problem when I want to start my releases while developing - not installed anywhere, just in _build/.... E.g. I run _build/default/rel/my_rel/bin/my_rel console an that succeeds, but when I run _build/default/rel/my_rel/bin/my_rel upgrade 0.1.1 it says my_rel-0.1.1.tar.gz is missing even though tarball is in _build/default/rel/my_rel.

Dunno how often do people start releases via rebar3 _build folder, but I find it convenient from time to time.

I misunderstood --output-dir before and I agree that it works properly. But I still think that it would be nice if tarball could go into releases or relases/<vsn> and that would enable upgrade/downgrade/install/unpack commands to be correctly performed in development.

P.S. sorry for close-reopen, hit my touchpad...

[ferd]

Hm. I don't remember if this is a path issue, or an Erlang issue. There's an old bug in Erlang/OTP where if you started your release without a tarball first, it never really picks up on releases fine or has issues with paths in some places: https://github.com/erlang/otp/issues/4041

Otherwise yeah I'd imagine just putting the tarball in the right directory would work to later call a relup. I've never really spent the time of doing it right in the _build directory though.

[MarkoMin]

Hm. I don't remember if this is a path issue, or an Erlang issue.

I would say that this is indeed path issue because it is resolved when I manually copy tarballs into releases or releases/<rel_vsn> directory.

I've never really spent the time of doing it right in the _build directory though.

Yeah, I find it particularly useful when experimenting with releases/upgrades. It gives you a sandbox right in your _build dir. It just saves your time - you don't need to install it somewhere if you want to try something. I usually do the following:

git checkout <prev_vsn_tag>
rebar3 tar
git checkout <HEAD>
rebar3 tar

and then you have your last 2 release versions installed in your _build and you can try to upgrade them.

2 solutions to this (if you agree that this should be taken care of) are:

1. Add current directory in search path for install, upgrade, downgrade and unpack commands in start script

• should cause minimal problems in backward compatibility, but this change is affecting part of the production (start script))

2. Put tarballs in /releases or releases/<vsn> directory

• could maybe cause backward compatibility issues (only in development!), but is not affecting start script

I'm for opt 2.

[ferd]

A third option is to just use a hook to make it work:

{post_hooks, [
  {tar, "mv $REBAR_BUILD_DIR/rel/YOUR_RELNAME/*.tar.gz \"$REBAR_BUILD_DIR/rel/YOUR_RELNAME/releases/\""}
]}.

which will just take the artifact and move it over.

I'm not sure opt 2 is a good idea (possibly breaking backwards compat with a bunch of CI scripts for people uploading artifacts is not a great plan). I think it would be safe tarball-wise (the systools docs make it clear we need to pass any extra file we want packaged).

Opt 1 is possibly safer, but the current directory wouldn't necessarily be a good plan unless it's appended at the end of all search paths to avoid suddenly favoring files outside of where they'd be expected, but as you said it does impact prod-level scripts.

[MarkoMin]

A third option is to just use a hook to make it work:

Yup, that works! I tried something like that before, but I wasn't satisfied because I didn't know about $REBAR_BUILD_DIR variable... Now I see that there is a lot of vars exported for hooks, which I believe is worth to be in the hooks documentation.

I'll close this issue because it's now solvable via hooks, and I'll try to enhance rebar3 hooks docs :D

Many thanks for the help!