Move to rebar3 [JIRA: CLIENTS-530]

[sdebnath]

Rest of Erlang world is moving to rebar3 to take advantage of the latest developer friendly features. Basho's riak client currently does not build with rebar3 because of protobuffs library (and potentially others). This is a problem for developers and customers as we are stuck with having to continue using older versions of the rest of the dependencies to make sure they can continue to use Basho's software. Which means missing out on bug and security fixes.

Rebar2 is also not actively developed any longer. Rebar3 is actively developed, has more repeatable builds (so they don't randomly break over time), support for packages, is more newcomer-friendly (on-boarding new employees with rebar 2.x is always a poor experience and always ends up in Makefiles to get things to work). There's better support for environment-specific or purpose-specific configuration as well, thanks to profiles.

It's time to get the erlang client up to 2015 :)

[mbbroberg]

FYI @cuyler @zeeshanlakhani @macintux.

@sdebnath - we were talking about this internally and there are some concerns to discuss on this thread. I'd like to understand what branch is best to work on before anyone puts in too much effort.

[danieldreier]

@mjbrender any updates on this? I've got a new project that would be a good candidate to experiment with this, but I'm quite hesitant to lock myself into it when it means using an older version of the tool chain.

[mbbroberg]

@danieldreier the scope is yet to be defined and that's the main concern. I can't offer an idea to getting this done until that is done. Sorry I don't have better news just yet - this code is an intimate part of core product code. // @sdebnath

[cuyler]

I've done some preliminary work on it in the last couple of days. There are some challenges with protobufs that we're ironing out. Not sure how long it will take but I'll drop some notes here as we progress. Thank you for your patience!

[sdebnath]

Appreciate it guys, this will be much welcomed by the community!

[danieldreier]

thanks @cuyler! I appreciate the update.

[cuyler]

Just a quick update here; We're in the process of writing some custom plug-ins for rebar3, due to the complex nature of how riak_pb works. We've run into a few issues with rebar3 itself, including this issue: https://github.com/rebar/rebar3/issues/731

We'll update as we make more progress.

[danieldreier]

I really appreciate the update and the work you're doing on this. I've been holding off on integrating with any of the basho erlang tools primarily due to wanting to use rebar3, so it's very encouraging to see periodic updates.

[cuyler]

OK! More to update. We've completed building a new rebar3 plugin, to help with the generation of protobuf definitions and now we're working on ( what we hope! ) is the final step. A new rebar3 protobuf compiler plugin. Unfortunately, rebar3 does not include a protobuf compiler by default ( as rebar does ) so we have to implement a new plugin to help us along.

We have a new repo here where you can track progress, should you desire. Right now, it's just a skeleton of a plugin, but we're working on porting/reshaping the protobuf code from rebar into it.

Apologies for this taking so long... It seems that this was much more of pandora's box than we realized... All work we needed ( and wanted ) to do, so... onward!

We'll update you again as we get closer to the compiler plugin working. Thanks for your patience!

[mbbroberg]

Hey @cuyler - curious if you could add an update when time permits! Thanks for your work and thanks to @sdebnath for keeping this dependency a priority!

[sata]

Any specific reason for going with Rebar3 instead of erlang.mk for the riak erlang client library?

What are the requirements for riak_pb that forces one to choose Rebar3 over erlang.mk?

I think it's worth understanding what Riak customers are using for build tools in their products before making any assumptions. Not everyone want to adopt Rebar3 nor ever will.

[sdebnath]

@sata Riak codebase primarily uses rebar as their dependency mgr, build and unit test tool and have been since they first released (AFAIK). Transition to rebar3 was a natural step to allow folks to continue using the latest rebar with their software projects. This allows customers, like myself, to upgrade all my software dependencies to their latest versions and get the latest security and bug fixes and not have to keep them downgraded due to the requirement of having to stick with rebar v2 for riakc.

The issue with community members doing the rebar3 PR was that there were complications with the pb build and needed folks with extensive knowledge in the area to get it done. Hence folks at Basho had to jump in. That being said, erlang.mk is picking up some steam, particularly with folks using cowboy as one of their core apps. If someone is willing to start the work, I am sure folks can help and can get it merged in.

[mbbroberg]

+1 to the above. You're on a thread with known Riak customers and Basho engineers, so it's safe to assume we have our customer's best interests in mind.

Hey @sata - glad to have you as part of the conversation. If you have a technical preference for erlang.mk over rebar3, could you explain why? It could inform the decision.

If there's a technical reason not to adopt rebar3, then there's also room to open issues to address the specifics. Whatever we discuss here, let's keep it technically sound and helpful.

[essen]

Erlang.mk author here.

The Riak client already works with Erlang.mk (no changes required, most of the Basho projects are tested on a regular basis (soon, daily) to be compatible with Erlang.mk). It will continue to be compatible regardless of your build tool of choice.

Today you have a tool that just works (Rebar), an alternative that just works, by patching Rebar out (Erlang.mk) and an "upgrade path" (Rebar3 is really a new tool with the same name) that doesn't work with your project. Moving to the latter sounds dubious at best.

Erlang.mk can generate rebar.config files automatically for backward compatibility. I'm perfectly fine with augmenting this to make the rebar.config file compatible with Rebar3 once it stabilizes. I'm also available if you have questions. I haven't made a migration tutorial to move from Rebar to Erlang.mk yet, but could if there's interest.

It's also worth pointing out that Erlang.mk is tested on Linux, OSX, FreeBSD and Windows (there will be CI of all of these and more very soon, I got sponsoring for that).

Of note, RabbitMQ is moving to Erlang.mk. The migration is in the final phases and should be complete sometimes this month. Also, @lehoff started some work to port riak_core to Erlang.mk (https://github.com/basho/riak_core/tree/th/erlang.mk). Note that riak_core works fine with Erlang.mk today, by patching Rebar out automatically.

Note that I have purposefully ignored the question "why one over the other" as I believe it's not for me to say. :-)

[danieldreier]

@essen thanks for chiming in. I've heard quite a few people in the erlang community speak favorably of erlang.mk. I hadn't thought to try it with the basho projects -- as far as I'm concerned that basically solves my use case for this issue, which is that I didn't want to be tied to rebar 1 by depending on this project. It sounds like the easiest way to achieve that is using erlang.mk.

[fp]

After having spent a bit of time digging through this, finding the hex riakc package broken, because it doesn't actually build the all of the riak_pb/src/*.proto files. This is even using rebar3 overrides for the riak_pb rebar.config and @tsloughter riak_pb_msgcodegen. At that point, I gave up, since I don't have the will to figure this out any further... I did just wanted to use riakc in my project.

I ended up with the following in my makefile, which allowed the riakc dependency in rebar.config to "just work":

## This is a bit of a mess, but it is required to get riakc to compile with
## the protobuff codecs built and included... because rebar3 riakc does not
## propertly pull in a working riak_pb build. So we basically build riakc
## twice -- once externally to get riak_pb built, and then the second (real)
## time using the rebar3 package by using the riak_pb that was built in the
## first go.
_checkouts/riak_pb: _checkouts riak-erlang-client
    (cd _checkouts && ln -sf ../../riak-erlang-client/deps/riak_pb .)

riak-erlang-client: ../riak-erlang-client
    make -C ../riak-erlang-client deps all

../riak-erlang-client:
    (cd .. && \
    git clone https://github.com/basho/riak-erlang-client.git)

_checkouts:
    mkdir _checkouts

Hopefully, that helps someone else trying to pioneer in rebar3 territory.

[alphashaw]

I just ran into the same probem

[lucafavatella]

I put together a minimal rebar3 demo application with riak_pb as a dependency, using the erlang_protobuffs rebar3 plugin by @cmkarlsson. It looks working. A similar approach could be used in riak-erlang-client.

[lucafavatella]

I had not realized that @tsloughter had already posted the full workaround for building riakc with rebar3.

I found that because the riak_pb_messages module was not getting generated, as the riak_pb_msgcodegen was needed. I found two versions: one incomplete, one working.

I also updated the minimal rebar3 demo application with riak_pb as a dependency (specific commit) - if of any use to anyone attempting to build riakc with rebar3.

[goncalotomas]

Any news for this issue?