[WIP] nowmix: use a Mix-pleasing version constraint format

[fenollp]

Continuation of https://github.com/certifi/erlang-certifi/pull/25 which itself is a solution to https://github.com/benoitc/hackney/issues/484 Discussion from https://github.com/certifi/erlang-certifi/pull/25#issuecomment-369590347 So this PR is looking to please rebar3, Mix & erlang.mk dependency handling.

It seems the ~>X.Y.Z or ~>X.Y format pleases both Mix and rebar3. However erlang.mk does not like this format. @essen Do you think it would be reasonable for erlang.mk to support ~>X.Y.Z as if it was just X.Y.Z? I am not even sure that is acceptable by Mix! Otherwise I don't see an easy fix to this issue and that makes using BEAM libs across languages not easy, at least if erlang.mk is concerned.

Note: erlang.mk fails to build with

make[1]: Leaving directory '/home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/rebar'
/home/pete/wefwefwef/erlang-certifi.git/bla
rm -f /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/deps.log
rm -f /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/apps.log
mkdir -p /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk
set -e; for dep in  ; do \
    mkdir -p $dep/ebin; \
done
set -e; for dep in  ; do \
    if grep -qs ^$dep$ /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/apps.log; then \
        :; \
    else \
        echo $dep >> /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/apps.log; \
        make -C $dep IS_APP=1; \
    fi \
done
mkdir -p /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk
set -e; for dep in /home/pete/wefwefwef/erlang-certifi.git/bla/deps/certifi ; do \
    if grep -qs ^$dep$ /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/deps.log; then \
        :; \
    else \
        echo $dep >> /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/deps.log; \
        if [ -f $dep/GNUmakefile ] || [ -f $dep/makefile ] || [ -f $dep/Makefile ]; then \
            make -C $dep IS_DEP=1; \
        else \
            echo "Error: No Makefile to build dependency $dep." >&2; \
            exit 2; \
        fi \
    fi \
done
make[1]: Entering directory '/home/pete/wefwefwef/erlang-certifi.git/bla/deps/certifi'
if test -d /home/pete/wefwefwef/erlang-certifi.git/bla/apps/parse_trans; then echo "Error: Dependency" "parse_trans (parse_trans)" "conflicts with application found in /home/pete/wefwefwef/erlang-certifi.git/bla/apps/parse_trans." >&2; exit 17; fi
mkdir -p /home/pete/wefwefwef/erlang-certifi.git/bla/deps
mkdir -p /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/hex /home/pete/wefwefwef/erlang-certifi.git/bla/deps/parse_trans; curl -Lfo /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/hex/parse_trans.tar  https://repo.hex.pm/tarballs/parse_trans-~>3.2.0.tar; tar -xOf /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/hex/parse_trans.tar contents.tar.gz | tar -C /home/pete/wefwefwef/erlang-certifi.git/bla/deps/parse_trans -xzf -;
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403 Forbidden
tar: /home/pete/wefwefwef/erlang-certifi.git/bla/.erlang.mk/hex/parse_trans.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
/home/pete/wefwefwef/erlang-certifi.git/bla/erlang.mk:4813: recipe for target '/home/pete/wefwefwef/erlang-certifi.git/bla/deps/parse_trans' failed
make[1]: *** [/home/pete/wefwefwef/erlang-certifi.git/bla/deps/parse_trans] Error 2
make[1]: Leaving directory '/home/pete/wefwefwef/erlang-certifi.git/bla/deps/certifi'
erlang.mk:4311: recipe for target 'deps' failed
make: *** [deps] Error 2

PROJECT = bla
DEPS = certifi
dep_certifi = git http://github.com/fenollp/erlang-certifi nowmix
include erlang.mk

cc @ericmj cc top @github/erlang/rebar3: @tsloughter @tuncer @ferd

[essen]

Erlang.mk will need to be patched. I'm not familiar with this version syntax thing but it can probably be handled easily during autopatch.

[fenollp]

Cool so since erlang.mk doesn’t fetch/parse/talk to hex.pm to discover the versions of a package that exist then a constraint format that would satisfy all 3 tools is "~>X.Y.Z".

This format is accepted by rebar3 (so says CI). Erlang.mk would need a simple patch to check format then drop ~> before fetching package. Now I don’t know if Mix handles this format (the Z part may be an issue). Can anyone try it? Thanks

[ericmj]

Mix accepts the ~> x.y.z format but the correct way to specify the requirement would be ~> x.y according to semver.

[tsloughter]

Rebar3 hex will switch to default ~> x.y when unspecified.

Best to do the same in certifi. If erlang.mk wants to support dependencies that use hex deps it can do so by resolving them like the rest of the clients. Or if erlang.mk really didn't want to support hex it could parse the rebar.lock file for the full version string, it is expected users commit those, though it isn't guaranteed.

[essen]

Sounds like rebar.lock is the way to go. Thanks for the tip.

Like I've said in the past, I don't want to write a third incompatible hex package manager. We already have enough issues with different versions being selected by rebar3/mix when working on an Elixir project with mix and then depending on it via rebar3. If we had an actual package manager for the whole ecosystem erlang.mk would already use it. For the time being I'm settling for maintaining compatibility with existing projects.

@fenollp Can you open a ticket? And as a workaround for now you don't need certifi to have dependencies specified as x.y.z or ~>x.y.z, you can just add the version you want as a BUILD_DEPS of the top level project and we can work on a fix later.

[tsloughter]

Rebar3 and Mix resolve dependencies differently. There can not be a tool that gives the same result to be used by each of them, they are not and will not be compatible, so this single "package [resolver]" will never exist.

[fenollp]

Yeah well, this issue doesn’t concern me: I use rebar3. So I’m not pushing a Makefile to certifi that imports erlang.mk and duplicates the versioned deps. To me the most simple way to make everyone happy here is the 2 line fix in mk to handle ~>X.YZ. I am not opening a ticket come on

[benoitc]

Hrmm, just to confirm then both rebar3 and mlix able to resolve a dependency using such format {"parse_trans, "~>3.2"}, (@ericmj @tsloughter ) but erlang.mk isn't (@essen ) ? That starts to be complicated... At least even if the resolving is different I would expect that all tools could handle the hex.pm api and package name resolving the same way :/

I agree that the patch from @fenollp is the correct solution for now, it should handle the initial issue spotted by @ericmj, isn't it? Question now is if it can work with erlang.mk .Another way is to just resolve on ~>3.2 and add a rebar.config.script for erlang.mk to override the dependency. @essen is there a way to detect erlang.mk is used from the environment?

[benoitc]

Thinking more of it @essen if we publish with the rebar.lock would it work? Would erlang.mk uses it to fetch the dependencies? Does it execute the rebar command for it?

[benoitc]

I'm tempted to revert to our own custom parse transform and keep the deterministic flag to make it simpler for all including that lib already. That change appear to not be that easy to be handled in a timely fashion. @fenollp what would we lost in doing it?

[essen]

Yeah well, this issue doesn’t concern me: I use rebar3.

Ah sorry looks like I missed the original thread. I'll forward my comments there.

@benoitc Don't worry about it, there is a very easy workaround so just do the ideal changes for your usage and I'll make it work again.

[ericmj]

Like I've said in the past, I don't want to write a third incompatible hex package manager. We already have enough issues with different versions being selected by rebar3/mix when working on an Elixir project with mix and then depending on it via rebar3. If we had an actual package manager for the whole ecosystem erlang.mk would already use it. For the time being I'm settling for maintaining compatibility with existing projects.

We are working on an erlang library with shared functionality for implementing Hex package managers: https://github.com/wojtekmach/hex_erl. We are starting out with support for the HTTP API, building/unpacking/verifying package tarballs, and fetching/reading the registry. We will add more to this library as time goes on, for example the dependency resolver algorithm. But as @tsloughter said it may be that you don't want to use the same resolution algorithm that mix/hex does so the library will be plug-and-play where you can chose to use the parts that you want to use. I'm not sure how an erlang library will work with erlang.mk since it's based on makefiles but there's probably something we can figure out, maybe give it an escript frontend you can call.

Unfortunately there is not a single package manager we can build that will work for all build tools since their dependency handling works completely differently.

Hrmm, just to confirm then both rebar3 and mlix able to resolve a dependency using such format {"parse_trans, "~>3.2"}

Yes, this format will work for both rebar3 and mix and it's the format that is best suited for semver.

[benoitc]

@fenollp can you update the PR to use {"parse_trans, "~>3.2"} ? I will do a release right after and document it.

[essen]

I'm not sure how an erlang library will work with erlang.mk since it's based on makefiles but there's probably something we can figure out, maybe give it an escript frontend you can call.

Like any other Erlang libraries. You don't need to do anything, it's trivial for erlang.mk to use Erlang code. The autopatch functionality for rebar compatibility is written in Erlang and uses some code from rebar2 for example. This would be no different, so don't worry about it.

Sounds good. I will happily use this along with the dependency resolver algorithm in Erlang.mk once it's available. I'm guessing I can easily make it so that hex dependencies follow the hex rules, and other dependencies (git, etc.) can continue following what Erlang.mk is doing (first fetched wins).

All tools not having the same rules is unfortunate to say the least but this is a step in the right direction and at least this will mean that someone using Erlang.mk with only hex deps will end up with the same dependencies as someone using mix, so that's already something.

I hope there'll be some kind of announcement once the dependency resolver bits are in. Cheers.

[fenollp]

rebar3 appears to be handling tildes the same as Mix. Also now that I read the docs on ~>, ~>X.Y.Z is too restrictive. There's apparently a workaround for erlang.mk projects and there will be a patch in the future, so I'm just going to update PR to say ~>X.Y.

[fenollp]

Done. BTW @benoitc could you activate TravisCI for this project? EDIT: then we could setup a Travis cron task to build master!

[benoitc]

@fenollp thanks. CI is enabled now: https://travis-ci.org/certifi/erlang-certifi/builds/348220536

[fenollp]

Thanks!