master is broken

[Cloven]

[11:45:09] fsg:p $ git clone https://github.com/erlware/relx Cloning into 'relx'... remote: Counting objects: 2889, done. remote: Total 2889 (delta 0), reused 0 (delta 0), pack-reused 2889 Receiving objects: 100% (2889/2889), 4.47 MiB | 2.14 MiB/s, done. Resolving deltas: 100% (1773/1773), done. Checking connectivity... done. [11:45:54] fsg:p $ cd relx [11:45:55] fsg:relx git:(master) $ ls CONTRIBUTING.md README.md examples rebar.config rebar3 src LICENSE.md bootstrap.cmd include rebar.config.script rebar3.cmd test PKGBUILD cover.spec priv rebar.lock relx.config [11:45:55] fsg:relx git:(master) $ ./rebar3 compile ===> Verifying dependencies... ===> Fetching getopt ({pkg,<<"getopt">>,<<"0.8.2">>}) ===> Moving checkout "/tmp/.tmp_dir293196315253" to "/Users/fsg/p/relx/_build/default/lib/getopt" ===> Package not found in registry: erlware_commons-0.15.0

[tsloughter]

You should just have to update your package registry: ./rebar3 update

[Cloven]

That works. Recommend checked-in master has a rebar3 that's already updated, if rebar3 is really going to be a dependency here.

[tsloughter]

@Cloven it isn't rebar3 that is updated, it is the package index on disk. This is like an apt-get update.

[Cloven]

where is this package index located? Not in the repo, but somewhere else, globally?

[tsloughter]

Right, default is at: ~/.cache/rebar3/hex/com/amazonaws/s3/s3.hex.pm/tarballs/packages/registry

[Cloven]

rebar3 is the new systemd. Gotcha.

[tsloughter]

Sure.

I don't know how a package index can be contained in the client's repo as people wouldn't be able to publish their packages to it.

[Cloven]

People can't publish their packages to a binary file located at, no kidding, ~/.cache/rebar3/hex/com/amazonaws/s3/s3.hex.pm/tarballs/packages/registry either.

[tsloughter]

Correct, that is a local cache of the registry so the network doesn't have to be hit unless the user wants or needs to update it.

[Cloven]

I guess my point is that putting a random binary file in a hidden directory 10 levels deep in a user's home directory, in which the first part is named inocuously and is likely to collide with other uses of that name; the second part is named for the product; the third part is named after the barename of an elixir package manager; the fourth part inherits from java jargon; the fifth part is named for an internet company and their particular platform as a service; the fifth part is named for one of the services they offer on that platform; the sixth part is named more completely for that same aforementioned implementation of a package manager in the elixir language; the 7th part is named 'tarballs', and you'd think it contains tarballs, but no, it contains only: the 8th part, which is a directory named 'packages' which nevertheless contains not just tarballs, but also a bunch of binary ets tables, is beyond poor taste and into the world of utter total comedic ridiculousness.

I don't expect rebar3 or any other build tool to store a bunch of tar files in a hidden directory off my home directory named after a random bunch of irrelevant words. I expect dependencies to be situated in the development environment for which they're relevant. What could possibly have led to a universe in which rebar3 developers could believe otherwise?

[tsloughter]

The directory name is the name of the index, this is to support multiple indexes. If we only allowed the default to be used we'd simply put it in ~/.cache/rebar3/packages. And it has the confusing tarballs dirname in it because the CDN url is literally s3.amazonaws.com/s3.hex.pm/tarballs. Suggestions welcome.

We believe in having a cache of packages because hitting the network to download packages you've already downloaded is annoying to us. This is fairly common practice.

Cabal: ~/.cabal/packages/hackage.haskell.org/ Opam: ~/.opam/repo/default/archives

The Opam use of default is something we may copy instead of the long url name.

[Cloven]

.rebar3.conf:

{rebar3_opts, [ {cache, false}, %% default {package_cache, "~/.rebar3_package_cache"} ] }

package_cache directory structure:

{repository_source}/ tarballs/ dependency_db/

e.g.

hex/ tarballs/ whatever.tar etc.tar dependency_db/

[ferd]

So the reason for using ~/.cache/rebar3 is that ~/.cache is relatively common of a place to store stuff that is a cache and safe to delete (making it easier than combing through ~/.rebar3 for the same stuff).

The reason for rebar3 as a subdir is that we can cache other things, such as dialyzer base PLTs so that checks and builds are faster in the future. If we followed your scheme, we might need ~/.rebar3_package_cache, ~/.rebar3_dialyzer_cache, and so on for every command that may end up storing data. That would probably pollute up your home directory more than the current scheme.

The repository source in this case is the index URL, and that's what gives you com/amazonaws/s3/s3.hex.pm/tarballs/packages/registry The only major difference is probably that we have the order reverted. tarballs/ must be within the repository source, because it's easy to foresee tarballs with the same names being different in different indexes. Not doing so would expose us to conflicts where the cache gets poisoned by the app itself as it fetches app cowboy from a corporate private index into the cache where public cowboy is also read from.

I do see value in letting people disabling the cache mechanism altogether, though I'm not sure I'd set it to false by default.

[tsloughter]

Note that ~/.cache/ specifically is used to follow xdg base directory specs.

[tsloughter]

Yea, it definitely will be on by default. But being able to turn it off is fine.

[Cloven]

I personally don't understand the reason for wanting to cache packages on a per-user basis even a little. With all of the build solutions, packages are downloaded once, when 'make deps' or 'rebar3 whatever' runs, and then they are in the project, never to be redownloaded again. Certainly I don't want a bunch of random uncompressed tar files in random places on my disk every time I load up a new project; given that most projects will version pin if they're sensible, I'd end up with ~ M x N tar files. What benefit does that bring anyone, by default?

[ferd]

They're cached so the next time you have another project that needs lager, cowboy, folsom, relx, whatever, you copy it from the cache rather than hitting the network every single time.

We're kind of assuming that generally, Erlang developers have more than one Erlang code base. If you're looking to ship code to users permanently, releases are the correct tool and won't require ever hitting a cache.

[tsloughter]

The actual Erlang app tarball within the package tar is compressed.

Again, this is common practice and we do it per-user simply to not require access to a shared directory.

[Cloven]

@ferd -- each of my erlang projects has a different version of lager, cowboy, folsom, and relx (and otp, and). Keeping old versions around, outside of the repos they live for, is still not sensible. With one day's use, already I have two versions of 'bbmustache' ready to confuse the fuck out of me next time I use find.

@tsloughter -- wrong on the first count:

tar tvvf erlware_commons-0.13.0.tar -rw-r--r-- 0 0 0 1 Jun 22 14:08 VERSION -rw-r--r-- 0 0 0 64 Jun 22 14:08 CHECKSUM -rw-r--r-- 0 0 0 1122 Jun 22 14:08 metadata.config -rw-r--r-- 0 0 0 45860 Jun 22 14:08 contents.tar.gz Archive Format: POSIX ustar format, Compression: none

and I was unable to find another language or packaging system that took random dumps in my home directory. Out of 17, GHC comes the closest; it stores textual configuration files relevant to the packages it has downloaded. So it's not "common practice" either. Not sure what you're on about.

[essen]

Out of curiosity, if you follow XDG for caching, why not follow XDG for the ~/.rebar3 folder? Depending on what you store, it should be ~/.config/rebar3 or ~/.local/share/rebar3, not ~/.rebar3.

$XDG_DATA_HOME defines the base directory relative to which user specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.

See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html

Half-following XDG makes no sense whatsoever since you're still polluting $HOME.

[liveforeverx]

hex have $HEX_HOME, mix have $MIX_HOME environments, if there no $REBAR3_HOME environment, it make sense to provide it. As for example, in different buildservers, somebody want to specify explicitly the cache directory(for example, by us by build server it is mandatory requirement not to polluting $HOME).

[essen]

I'm not sure how that's related to what I'm saying?

If $REBAR3_HOME is currently ~/.rebar3 by default, this default should be changed to either $XDG_DATA_HOME/rebar3 or $XDG_CONFIG_HOME/rebar3 depending on what this folder contains. Otherwise you're only half following XDG and are still polluting $HOME.

[liveforeverx]

@essen Not related. Have not known, that REBAR3_HOME exists.

Ignore my comment.

[tsloughter]

@essen it isn't ~/.rebar3 it is ~/.config/rebar3/

[essen]

Cheers.

[tsloughter]

@Cloven I said the Erlang app bundle was compressed. Which it is. You are looking at the archive which contains the app tar.gz. Compressing it again makes no sense.

Also, I didn't mention GHC (which isn't a project build and packaging tool). I did mention Cabal, which as I already said does cache packages to ~/.cabal/packages/hackage.haskell.org/.

As does Opam to ~/.opam/repo/default/archives/.

Please read what I say before claiming it is wrong.

[tsloughter]

Forgot to mention Leiningen/Maven which go to ~/.m2

[Cloven]

On consideration, I think my criticisms have been less constructive than I would have liked. I apologize for giving any personal offense. So here's my constructive criticism.

In my opinion and in my personal history, rebar has been a problematic build tool. I've seen projects that required a specific rebar version; I've seen bizarre inexplicable dependency errors; I've seen it be very slow; I've seen it languish in apparent neglect; I've seen a lack of usability. In response to this, I eventually tired and went over to erlang.mk, the faults of which have been fewer in number from my perspective, solved faster, and solved in a tasteful manner. That is my personal background against which, wrongly or rightly, I look at rebar3.

When a user first clones relx, they get a rebar3 executable. When they, per the instructions, run 'rebar3 compile', they get an inscrutable error message. There is nothing in the error message, nor in the documentation of relx, nor in the 'basic usage' documentation of rebar3 at rebar3.org, nor in the output of 'rebar help' which led this fairly experienced developer to understand how to resolve the issue. So I posted an issue to Github.

What I would have expected: I would expect ./rebar3 compile to do whatever it takes, optionally with a user prompt to confirm an additional action, to actually compile the code. As a user of a build tool, I would expect rebar3 to say "dependency xxx not found, downloading", to download and compile, and to continue making progress.

It turns out that rebar3 is not just a build tool. It also wants to be a local repository of downloaded dependencies. To me, this is surprising. There are combo tools in the java family like lein and mvn which both build and try to keep a local repo, but this is because there can be several hundred packages taking up gigabytes or more involved in building, e.g. Apache Storm from scratch; it was driven by desperate enterprise necessity, not because it's tasteful. In my personal opinion which I claim as mine alone, there's no reason to copy that behavior in erlang.

What I would have expected: for a build tool to be just a build tool, barring any obvious need to be otherwise. To get that very solidly right, as Ant did in its day, before moving on to vastly more complex and fraught responsibilities, and then only when necessary. "cabal hell" and "maven sucks" are both well known bumperstickers for well-deserved reasons.

What else I would have expected: lein and mvn both feature write-through caching. If the local copy isn't there, fetch upstream, save it, and continue to build, transparently and seamlessly (maybe warning the user that files are being stored not in the project, but in a cache).

It then turns out that rebar3 stores this intermediate cache in, to me, a confusingly named string of directories. Perhaps, as with clojars or maven central or whatever, one could come up with a simple mnemonic name for these repos, and use the shortname. Certainly splitting the url by periods into subdirectories doesn't seem likely to be valuable.

What I would have expected: for caching to be off by default, because even very large erlang projects (e.g. riak) do not have the infinite-jars problem and downloading dependencies takes on the order of seconds or minutes, nor is there an enterprisy giant mountain of spring dependencies that must be included in every build that is updated slowly enough that keeping a local copy bears significant fruit.

What else I would have expected: instead of relying on an (in my opinion) confused non-unixy naming scheme from freedesktop.org, do as unix people have done for several decades and create a subdirectory named after the product so it's discoverable and individually actionable from your home directory (e.g., .vim, .cups, .bundler, .subversion, .netbeans, .git, .cocoapods, .m2, ... .heroku....

It then turns out that rebar3, when you, e.g., 'rebar3 compile', says 'verifying dependencies' and then 'compiling relx', and then exits with error code 0 to the shell. If you then type './relx', you get nothing. After looking around for it a bit, you discover that the product has been built to _build/default/bin. Looking around further, there's a _build/default/lib directory as well. There is no explanation of why 'build' has a prepending underscore; why there is a 'default' directory and what it might do; or what the lib directory is for. The lib directory appears to be a duplicate of the 'deps' directory which continues to exist and which also has beams.

What I expected: for all built executables to be built into the project root, as custom and history has taught is regular. Or, failing that, for a message to be emitted saying 'compiled successfully into: ...'.

What else I expected: if the tool is going to be enterprisy and start creating hiding spots for artifacts, then do so consistently. relx does not build to _build/default/_rel/; it builds a node in the top level directory of the repo, named _rel. That seems to make sense to me, although I've never understood why the underscores on these directories.

What else I expected: 'rebar3 install'. If you are building relx, you probably want to use it; so you probably want to install it somewhere in your path. Instead, you are left alone to copy it manually out of this hidden set of directories to your desired location.

Some positives I would like to finish with:

'rebar dialyzer' is good. Very good. It tells you what it is doing; it performs intermediate tasks if necessary and tells you about them; when it takes a big poop in your ~/.cache/.. directory, it lets you know in no uncertain terms ("Adding 156 files to "/Users/fsg/.cache/rebar3/rebar3_18.0_plt"").

The 'escriptize' format for escripts is way better and more sensible than, e.g., Loic's in erlang.mk. Although at first I was confused, the idea of keeping the paths the same for all applications in an escript is inspired, and appears to be more like what OTP intended. I hope to contribute that format back to erlang.mk in the near future.

I know you guys have labored hard at rebar3 for a while; I know how much it sucks to get straight up negative feedback on a free open source project. For my snippy tone earlier, I hope you accept my apology. I hope this more explicatory feedback is of some use, either as a guide to a more tasteful and well targeted tool, or as a dartboard in the kitchen.

F.

[tsloughter]

@Cloven thanks for this. You had useful feedback mixed in with your original complaints ;). And I've already acted on them. The relx README now contains ./rebar3 update, this was an oversight when replacing git with hex deps. And rebar3 will now mention running rebar3 update as a potential way to resolve a dependency that is unable to be resolved.

Including the path to the index that is written, similar to how the dialyzer plugin does it, is also a good idea and I plan to add this output soon.

The rest I mostly just disagree with. Polluting ~/ instead of placing in ~/.cache and ~/.config is something I find very annoying about tools that do it. Redownloading deps has always been an annoyance to me and others, which is why the cache was added and is on by default.

Now, the reason we don't fetch the dep even if it exists until you run rebar3 update is a tough one. Many tools don't require this. For example Mix will always update from the remote registry. Cabal, Opam and others (like OS package managers, apt/yum/etc) require the user to run an update manually instead of assuming they want it done. It is a behaviour we very well may revisit in the future.

Oh, and the _ is to signal you do not commit it. And rebar3 uses _build so all artifacts are contained in a single location instead of splitting it out between deps, _rel (in rebar3 releases go to _build/<profile>/rel/, escripts written to cwd and ebin/. And putting the executable at the top level is not the custom or history, unless you are simply referring to rebar.

If you still have a deps/ and ebin/ it means you had previously built relx before updating a commit that uses rebar3.

And lastly... there is no rebar3 install because it is a developer's tool and not for installing end user programs. I despise when programs tell you to install with npm install <name> or similar. For relx that wouldn't be an issue since it is a developer tool as well, but it is a bad precedent to be installing executables at all.

[ferd]

Yeah I have to echo the sentiment, I'm very appreciative of the full explanation there, and I'll try to make improvements in line with the comments mentioned. Specifically mentioning where we cache packages and whatnot, where files might be created and so on so people don't search.

To add to the comments on _build/default/..., one of rebar3's core features is profiles, letting you have specific sets of dependencies and options according to the type of activity you do. Test runs will use a test profile. Building a release in default mode might use symlinks while prod profile would create a standalone build with OTP copied. You could have specific C options in a windows profile, such that you could have a _build/ directory with entries for test, prod, default, windows, windows+prod, windows+test for example. All common dependencies in default are symlinked to the other profiles (or copied from the cache if there's no symlinks and packages are used).

But the nice thing is that over time, we'll stop having meck and proper shipped in production systems while still making them generally available to projects.

I'm not sure how we can make this more intuitive, though. Maybe just outputting "artifacts will be created in _build/default/" could help enough, maybe we need to do better.

I realize that this is confusing when you take a first run at it, but as a regular user, it has to make more sense than all artifacts from different runs with conflicting options getting smushed into the same directory.

In any case, thanks again for the full explanation, this is much appreciated.

E: edited quotes for github layout, sorry if you get duplicate emails.

[Cloven]

Guys, thanks for your responses.

On the forced manual './rebar3 update' -- although I agree that apt et al do that too, that's not a great reason to follow suit. Under what circumstances would the user ever get into a situation where they couldn't resolve a dependency, but then decided not to run './rebar3 update'? What benefit does it offer to rebar3 users to include that extra step, and what does it cost to just go ahead and skip it for them?

On polluting ~ -- while I understand the idea of disliking adding another dotfile, now that you've decided to hide your dotfile in a different inocuously named dotfile, you have created two problems and two different mental standards to keep track of. What will happen when you want a per-user file for customization of rebar3's behavior, for example if people want to turn off the package management features? Will that go in ~/.cache too, where it will doubtless be deleted by systemd periodically (half joke)?

On no 'rebar3 install' -- I understand the dislike, but again, now that you've deviated from the standard for every single similar build tool out there, and abandoned a half decade of 'make install' history, you have created two problems. How does an open source author shipping a tool handle this scenario? "ok, after you run './rebar3 compile, type 'cp -R _build/default/product/* /usr/local/bin', except wait, the config file goes in /usr/local/etc, and the data file needs to be in its own directory which I'd like you to create in ...", and so on? They ship a custom script? With per unix defaults? ...

As far as using _build and intending the special underscore to mean 'do not commit' and all of this profile stuff of which I was blissfully unaware and 'default' and so on -- I think this points to the key underlying muddle. Rebar3 is, in my opinion, overwhelmingly likely to be used as a compilation tool, as is Make/erlang.mk and rebar. My estimate is that 99.9% of the time, maybe a lot more, it'll be used to compile for the native local platform in release configuration mode, and optionally to run tests. For that set of users, _build/default/whatever/ is inscrutable enterprisy release engineer devops klingon. Heck, for me, it's IEREDK; what would a profile of 'windows' on my freebsd box even contemplate doing? When would I ever have two profile directories filled at the same time? What's a testing profile for when I can just do 'make all eunit ct' or the like?

Now -- maybe it's useful for you guys; maybe you occupy a rarefied cloud of crazy ass inscrutable enterprisy release engineer devops nirvana; and I suspect that in fact, you may very well. But in making the tool suit your weird, probably quite necessary and important but weird, theories about artifact building, you have, in my personal opinion, created some pretty large holes in the sweet spot for the 99%. If you are to try to become the standard build system for erlang, which I believe is an important goal, then I would suggest you reevaluate rebar3 through the lens of its use cases and predecessors; innovate cleverly far less; swallow pride re: doing things the normal unix way; and make the IEREDK a configurable option rather than an inscrutable default.

Anyway -- more ignorable rambling; appreciate all that you do, even if I don't always agree with it.

F.

On Sat, Aug 22, 2015 at 9:39 PM, Fred Hebert notifications@github.com wrote:

Yeah I have to echo the sentiment, I'm very appreciative of the full explanation there, and I'll try to make improvements in line with the comments mentioned. Specifically mentioning where we cache packages and whatnot, where files might be created and so on so people don't search.

To add to the comments on _build/default/..., one of rebar3's core features is profiles, letting you have specific sets of dependencies and options according to the type of activity you do. Test runs will use a test profile. Building a release in default mode might use symlinks while prod profile would create a standalone build with OTP copied. You could have specific C options in a windows profile, such that you could have a _build/ directory with entries for test, prod,default,windows,windows+prod ,windows+testfor example. All common dependencies indefault` are symlinked to the other profiles (or copied from the cache if there's no symlinks and packages are used).

But the nice thing is that over time, we'll stop having meck and proper shipped in production systems while still making them generally available to projects.

I'm not sure how we can make this more intuitive, though. Maybe just outputting "artifacts will be created in _build/default/" could help enough, maybe we need to do better.

I realize that this is confusing when you take a first run at it, but as a regular user, it has to make more sense than all artifacts from different runs with conflicting options getting smushed into the same directory.

In any case, thanks again for the full explanation, this is much appreciated.

— Reply to this email directly or view it on GitHub https://github.com/erlware/relx/issues/384#issuecomment-133781101.

[tsloughter]

What will happen when you want a per-user file for customization of rebar3's behavior, for example if people want to turn off the package management features?

That goes in ~/.config/rebar3/rebar.config.

How does an open source author shipping a tool handle this scenario? "ok, after you run './rebar3 compile, type 'cp -R _build/default/product/* /usr/local/bin', except wait, the config file goes in /usr/local/etc, and the data file needs to be in its own directory which I'd like you to create in ...", and so on? They ship a custom script? With per unix defaults? ...

No. You would not install artifacts from compile. You build releases and distribute those, which many have take the form you describe (./bin, ./etc, ./erts, ...) so that unpacking in /usr/local puts everything in the right place.

What's a testing profile for when I can just do 'make all eunit ct' or the like?

For defining test only dependencies... I'm sorry you are overwhelmed by profiles but they are a very simple and useful addition to rebar. And I have no idea how it is related to the enterprise when, again, this use of a single output directory is common in most build tools.

I find "the unix way" often simply means, "the way I want/expect". We are attempting to build a modern Erlang build and packaging tool, learning from past Erlang tools as well as other modern languages/frameworks tools.

Now -- maybe it's useful for you guys; maybe you occupy a rarefied cloud of crazy ass inscrutable enterprisy release engineer devops nirvana; and I suspect that in fact, you may very well. But in making the tool suit your weird, probably quite necessary and important but weird, theories about artifact building, you have, in my personal opinion, created some pretty large holes in the sweet spot for the 99%. If you are to try to become the standard build system for erlang, which I believe is an important goal, then I would suggest you reevaluate rebar3 through the lens of its use cases and predecessors; innovate cleverly far less; swallow pride re: doing things the normal unix way; and make the IEREDK a configurable option rather than an inscrutable default.

Anyway -- more ignorable rambling; appreciate all that you do, even if I don't always agree with it.

F.