Add rebar3 support to YAWS build processes

[oubiwann]

Status: Proposal

Branch Location: https://github.com/erlsci/yaws/tree/rebar3-support

Branch Build Status: https://travis-ci.org/erlsci/yaws

Background: This has come up in two separate tickets, with a little conversation around them both that may be useful for context. In particular:

• 249 - Unable to use with rebar3 + LFE on OS X

• 254 - Unable to build with rebar3

Tasks:

• [x] Port YAWS' rebar script to rebar3
  • [x] Use rebar3 profile to support the ibrowse dep
  • [x] Update to use @blt's (hi Brian!) port compiler plugin
• [x] Decide on whether to pin deps to specific releases (for stability's sake) or continue pulling "master"
• [x] Update .travis.yml and test Travis CI builds
• [ ] Covert rebar templates to rebar3 templates
  • [ ] Create a new project: rebar3 new yawsnode testproj
  • [ ] Build a release of the new project with rebar3
• [ ] Update docs
  • [x] Update README.md to use rebar3
  • [x] Update doc/yaws.tex to use rebar3
  • [ ] Update www/rebar_release.yaws to use rebar3
  • [x] Update www/configuration.yaws to use rebar3
• [ ] Update tests
  • [x] Run locally on Linux
  • [ ] Run locally on Mac
  • [x] Run on Travis CI
  • [x] Run this branch as a dependency in another rebar3-based project and test YAWS functionality after that rebar3-based build
• [ ] Submit PR

Tasks that were proposed but decided not to do (at least, not in this branch):

• Decide on whether to support older rebar usage (newer rebar is broken) while also supporting rebar3
  • If both, provide a means of using the appropriate config file and
  • running the appropriate script (these could be fairly tricky and/or messy to do simultaneously)
• Possibly use a rebar3 profile to provide more of the YAWS_DEV_MODE functionality via rebar3

The following are no longer present, due to changes in the YAWS codebase since the creation of this ticket:

• Update test/support/include.mk to use rebar3

[vinoski]

@oubiwann: regarding your note in #254 about make (and assuming I understood it correctly): keep in mind that Yaws also provides a full automake-based build system, used by many/most of its long-time users. All the original rebar support was kept completely separate from the automake support, and rebar3 support needs to be kept separate as well. For that reason I don't like the idea of mixing make and rebar3.

[oubiwann]

@vinoski Got it, and good to know. That was for a separate project that had YAWS as a dep; it's not something I had (yet) considered for YAWS itself when adding rebar3 support ... and now the thought won't arise :-) Thanks!

[oubiwann]

Making good progress on this. Not too much to do for the most part -- rebar3 compile is working now, though there is more to checkout and update.

One note on testing so far:

• Tests are failing on R16B01 (Travis CI) though the issue doesn't seem to be rebar3 (The command "make distcheck -j4 V=1" exited with 2.)

I'll keep at it, but I'd saw the branch is ready for others to start testing rebar3 compile. I want to dig into the YAWS_DEV_MODE a bit more before actively seeking volunteers to seriously test ...

[vinoski]

Great, thanks for doing this.

Regarding Travis, we had a new commit in the past 12 hours and that build failed only on R16B03, but it passed just fine when I ran it with R16B03 in my own clone. We'll have to keep an eye on it.

[capflam]

This is a timing issue. By restarting the builds that failed, and not all builds, it works. A solution would be to rewrite tests that depend on a timer but this will be boring.

[oubiwann]

I've confirmed that rebar3-based projects which declare this branch as a dep will successfully build YAWS using only rebar3.

After I get a chance to make some other updates, I'll send out an email on the mail list asking for interested individuals to give the branch a go and report any issues.

[vinoski]

@oubiwann: looking good! I suspect getting the tests to work will take some effort, since we never got them working fully with rebar.

[oubiwann]

I've reconsidered adding rebar3 to the YAWS_DEV_MODE mix. Maybe in the future, but for now it distracts from the core purpose of this ticket.

[oubiwann]

Trying to get back to this. Just did a git rebase against latest master. Will update dev tasks in ticket description (above).

[expelledboy]

Does yaws have build support for rebar3 yet?

[vinoski]

Official support, no. But I know a number of users use rebar3 successfully with Yaws without official support, though I don't know the details of how they do it, and unfortunately none has offered to pitch in and help on this issue.

[leoliu]

For using Yaws from rebar3 as an embedded server it is mostly working except one thing yaws_sendfile. yaws_sendfile always starts yaws_sendfile_drv even if configured to use erlang's sendfile and so it will crash immediately because the port drv not compiling in rebar3. Making yaws_sendfile starts yaws_sendfile_drv on demand, or drop yaws_sendfile_drv and depend solely on Erlang's sendfile will help a lot.

[expelledboy]

So I managed to get it working I think, using the following rebar.config file.

{deps,
 [
  {yaws,
   {git, "https://github.com/klacke/yaws.git",
    {tag, "yaws-2.0.5"}}}
 ]}.

{plugins, [{pc, "1.6.0"}]}.

{overrides,
 [
  {override, yaws,
   [
    {plugins, [pc]},
    {provider_hooks,
     [{post, [
              {compile, {pc, compile}},
              {clean, {pc, clean}}
             ]
      }]
    }
   ]}
 ]}.

Hope others find this useful!

[vinoski]

Thanks @leoliu and @expelledboy, I'm sure others will find this information useful.

[expelledboy]

Perhaps you could add this to your top level README? With everyone moving to rebar3 this is probably the most pertinent question to a developer coming across yaws for the first time.

[oubiwann]

There's a branch linked at top of this ticket -- that branch works with rebar3. The guts are complete.

The reasons this work stalled out were all items in the tasks list not marked with a "check" in the ticket description ... the time needed to create new rebar3 templates for yaws was one of the biggest inhibitors.

I'll rebase the branch off of the latest master and push, so that it's up to date and ready to work on by other contributors.

Update: Rebased.

[vinoski]

Thanks @oubiwann!

[vinoski]

I've revived the original rebar3-support branch and pushed it here. It probably needs a little work, but hopefully we can bring it to master soon.

[jesperes]

What is the status of this now? I have some minor improvements to rebar.config.script and yaws.app.src.script (avoid regenerating files unless they have changed).

[vinoski]

I keep the rebar3-support branch up to date. There's still a lot of work to do on it before it can merge, though. Until then, feel free to submit a PR against rebar3-support.

[robertoaloi]

It would be really nice to get rebar3 support in master, since long-running branches are often painful to maintain. ~Anything special which is left to do? Anything we can help with?~ Just saw the issue task list 🤦

[oubiwann]

Hehe -- was just writing a response, and also just saw your edits ... so, nothing to add!

[oubiwann]

Well, maybe I can add something: I think the two most important things to do are probably updating the tests, and perhaps a moderately distant second, update the templates.

@vinoski can provide the definitive assessment 😄

[vinoski]

@oubiwann is correct — the tests are the biggest blocker, and the templates need work. The tests take advantage of running in an automake build environment, and so far I haven't figured out a way to either replicate that with rebar3 or rewrite it to take a different approach. And of course there could be other problems waiting once we get past that.