Feature request: a "Premake" build plugin?

[jjatria]

I've been looking into writing an Alien::Build distribution for a library that uses premake as part of its build process. While reading through the documentation for distribution authors I read about build plugins, and thought that a "Premake" plugin would be the way to go. But there is none in CPAN. So I thought I'd ask what you think about this as a contribution.

I hadn't even heard of the plugin system for Alien::Build until this happened, but I'd be happy to give this a try.

Some questions:

1. In the likely event that premake is not installed, is it the role of the plugin to get a copy it can use? They distribute precompiled versions for Linux, Mac and Windows, so maybe fetching a version to use would not be too crazy. Or should it just helpfully die with a message to the user?
2. Premake seems to be under heavy development, and the project I'm trying to use it for actually uses a versions that is still in alpha. How would you handle that?

Additional info

The library in question is Box2D. There is already an Alien::Box2D distribution, but it's outdated because it tries to download the sources from the defunct Google Code website. Looking into fixing that distribution is what sent me down this path.

[plicease]

I think (from minimum research on my part so correct me if I am wrong) that premake is a configuration / make file generator in the style of autoconf or cmake? If so then you want to construct something like Build::CMake or Build::Autoconf plugins. You may also want to have an Alien::premake as a separate dist to be used to provide a premake dependency in CPAN, a la Alien::cmake3. Alien::premake is the thing that should download the source and build premake if it isn't already in the PATH (or however it manifests itself when it is installed).

Consider if you want to provide Alien::premake. One reason that you might not want to is if the build process is too long (cpu) or large (disk) (someone once suggested Alien::gcc ... which is insane to begin with, but also would be a huge burden on cpantesters and end users). Another reason is it may be too complicated. I considered Alien::git for a while, and decided to punt and have it bundled with the git plugins and only work with the system git because the configuration and dependencies were too insane. If down the road alienizing Alien::git becomes easier we can enhance it to handle that. (though tbh I doubt that will happen).

The reason the build plugin should be in a separate distribution from the alien is that the plugin is always used by the alienfile, even when Alien::Box2D is doing a system install, but you only need Alien::premake for a share install. One of the design goals for Alien::Build was to aggressively reduce dependencies on system installs to make AB less onerous for Linux distribution packagers. Another reason you might want to put the plugin in its own dist is if premake itself and the plugin become stable and common enough in the future we can merge the plugin into the Alien::Build core. (This is what happened with the cmake plugin, which was its own distribution briefly while it proved itself). It sounds like we are a ways away from that for premake atm, but it is always good to keep that in mind.

Regardless of if you write an Alien::premake or not, the plugin should provide:

• helpers. %{premake} perhaps?
• default build hook, if that is reasonable (both the cmake and autoconf plugins have default build hooks).
• turn on DESTDIR if possible/advisable* and handle as appropriate

• set a share prereq on Alien::premake if appropriate

  * (some build systems use DESTDIR to allow you to set the prefix to the final destination but stage the install in a different directory for testing/packaging. This is beneficial if possible for packages that have paths compiled into their libraries or executables. There is some documentation for how it works in autoconf here: https://www.gnu.org/prep/standards/html_node/DESTDIR.html ).

Also after having read all of that big caveat is that the plugin author documentation is the most incomplete and may have a few errors here and there. I am available to answer questions (and fix bugs when it comes to that), but it may be a little frustrating at first. PRs on the documentation are obviously quite welcome.

From my perspective the most useful part of this exercise would be to have another set of eyes working in this space.

[plicease]

Also:

• the premake plugin should also set itself as a configure time prereq

[jjatria]

Thanks. This is all very helpful.

If looking at making an Alien::premake, it seems the easiest way would be to do so from what they call a "source package", which is to mean the source tree included in one of the tarballs that are released via the Github release mechanism. Is there an easy way to get to these? I first looked at the Download::Git plugin, but that looks for tags and clones the entire repository.

Github does provide an API endpoint for getting a JSON listing of the releases, with info about any linked tarballs. So maybe having a "Decode::JSON" plugin, or something like that, to extract relevant links from a JSON dump (which would make this not Github-specific) would be a good addition?

Incidentally, that would also be useful for the eventual update to Alien::Box2D...

[jjatria]

Or does Decode::HTML work with Github-style release pages as well? :thinking:

[jjatria]

As it turns out, it does work. But the version sorting is wrong:

Alien::Build::Plugin::PkgConfig::Negotiate> Using PkgConfig plugin: PkgConfig::CommandLine
Alien::Build::Plugin::Core::Download> decoding html
Alien::Build::Plugin::Core::Download> candidate *https://github.com/premake/premake-core/archive/v5.0.0.alpha4.zip
Alien::Build::Plugin::Core::Download> candidate  https://github.com/premake/premake-core/archive/v5.0.0.alpha4.tar.gz
Alien::Build::Plugin::Core::Download> candidate  https://github.com/premake/premake-core/archive/v5.0.0-alpha12.zip
Alien::Build::Plugin::Core::Download> candidate  https://github.com/premake/premake-core/archive/v5.0.0-alpha12.tar.gz
...

Is there a way to tailor the version sorting?

(Incidentally, even though this works, it only sees the releases on the first page).

[plicease]

For GitHub the Download::Negotiate plugin is usually the way to go so long as there are tags. The Git plugin is slow for large repos, as you noted. I have an issue open for this, along with some notes:

https://github.com/plicease/Alien-Build-Git/issues/1

I haven't gotten to it yet since as far as I know there isn't anything real that actually uses it, but I am pretty sure that it is doable from my notes. So many git based projects are on github and the Download::Negotiate plugin is lighter and more reliable.

Here is a snipit which shows a custom prefer hook that you can use:

use alienfile;

probe sub { 'share' };

share {

  start_url 'https://github.com/premake/premake-core/releases';
  plugin 'Download';
  plugin 'Decode::HTML';

  meta->register_hook(prefer => sub {
    my($build, $res) = @_;
    return {
      type => 'list',
      list => [
        sort { $a->{filename} cmp $b->{filename} } # some better sort here
        grep { $_->{filename} =~ /\.tar\.gz$/ }
        @{ $res->{list} },
      ],
    };
  });

};

I need to improve the Download::Negotiate interface to make this a bit easier (see #43 for my initial thoughts on this). Basically this works because without the version property set the download negotiator assumes that the start_url is the final url and it won't have to sort versions and doesn't load the Prefer::SortVersions plugin. That plugin as the name suggests uses Sort::Versions, but isn't IMO anything to write home about. It is right most of the time. This is a case where it isn't right.

Longer term once they get out of alpha you probably will be okay with the regular Prefer::SortVersions plugin.

Prefer wasn't my preferred name for this stage btw- but I wanted something that wouldn't conflict with the Perl built-in sort, which was my first choice.

[jjatria]

Thanks for the pointers. That actually works for this. I'll take a look at the other issues you pointed to and see if I can help.

In the meantime, I've put together a first kind-of working version of Alien::premake (baby steps). I ended up using Sort::Naturally for the version sorting (which I guess will become more reasonable once the current version stops being alpha), and that seems to work.

For the build stage, platform-specific files are in the build/gmake.$platform directory, and I wasn't sure if there was a reliable way to switch directories, so I used a subroutine to detect the platform and change directories (using File::chdir). I then wanted to use the Make plugin, but I couldn't figure out how to use it from within the subroutine, so I went ahead with a system 'make' call for now.

This sort-of works, but the included make files do not include a target for install, so I got stuck. So here are my questions about this bit:

1. Is there a way to reliably change directories (other than using Path::Tiny or something like it)?
2. Is there a way to use the plugin helpers from within a subroutine that is set as a phase hook? (Not sure if I used the right terminology here)
3. How do I proceed with installing a binary for which no install target is given?

[jjatria]

Incidentally, it turns out that the version sorting did not need to be that complicated. The reason the version numbers were getting out of whack is because the premake developers tagged some versions as premake-5.0.0.alpha and some as premake-5.0.0-alpha, so the problem was the dash versus the period.

In this case, a tighter filter was all that was needed in the end.

[jjatria]

And here's a first and untested version of Build::Premake, to at least have something to look at. Still on the TO-DO list is the handling of DESTDIR that you mentioned.

[plicease]

As of 1.06 you can actually cd in a command

https://metacpan.org/changes/distribution/Alien-Build#L90

and you can use the Alien::Build system method from within a hook to use helpers:

https://metacpan.org/pod/Alien::Build#system3

As for installing sans install target, there are %{cp} and %{make_path} helpers. Care has to be taken if you want to support Windows, in which case it is sometimes safer to use a code ref instead and use Path::Tiny instead. I am open to improvements in this area if we can figure out a good interface.

[jjatria]

Oh, thanks about the heads up about cd and system. I'll update the code to use those instead.

As for the install target, that sounds good. But I was wondering more about whether you knew of some good place to put the binaries( I guess this is where DESTDIR cones into play?).

For instance, if I install an Alien library using perlbrew, or some other local library solution, I would imagine the binaries would get placed somewhere where they don't need admin privileges. And I guess there is some internal way to figure this out?

And this gets more unclear under Windows (which of course it would be good to support if possible). Do you know of any good place to put them in this case? Maybe you've come across something similar in another if the Alien modules?

Sorry if these questions are a bit basic. This is all new territory for me!

On 14 Oct 2017 05:32, "✈ Graham ✈" notifications@github.com wrote:

As of 1.06 you can actually cd in a command

https://metacpan.org/changes/distribution/Alien-Build#L90

and you can use the Alien::Build system method from within a hook to use helpers:

https://metacpan.org/pod/Alien::Build#system3

As for installing sans install target, there are %{cp} and %{make_path} helpers. Care has to be taken if you want to support Windows, in which case it is sometimes safer to use a code ref instead and use Path::Tiny instead. I am open to improvements in this area if we can figure out a good interface.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Perl5-Alien/Alien-Build/issues/42#issuecomment-336609194, or mute the thread https://github.com/notifications/unsubscribe-auth/AC1o9ak3C8scj1MlNXQrhnKGVcHnmPYwks5ssDlXgaJpZM4P4W9X .

[plicease]

The questions are great actually, I will probably use this conversation as a reference to improve the documentation.

Executables should go in the the bin directory of your Alien's share directory ( %{.install.prefix}/bin ), much as the static libraries go into lib . The philosophy of the Alien system is that libraries and tools get installed in a place so they can be used by Perl, but not to replace the tools provided by the operating system. When you invoke requires in the aleinfile for a tool oriented Alien (any alien that installs stuff in the bin directory), this bin directory will be added to the PATH for the build. You can also use the tool from Perl by using the Env module. Eventually the Build::Premake plugin will do this. synopsis:

use alienfile;
share {
  requires 'Alien::premake';  # puts executables in the path during the build of a share install only
};

use alienfile;
share {
  plugin 'Build::Premake';  # this should presumably add Alien::premake as a share prereq
};

# from arbitrary Perl code:
# (you will rarely want to do this)
use Alien::premake;
use Env qw( @PATH );
unshift @PATH, Alien::premake->bin_dir;
system '...';

[jjatria]

I'm glad to hear this is helpful for you as well! I'll be happy to help improve the documentation if I can. After wrapping my head around the Alien::Build architecture, it turns out this is a lot easier to use and more powerful (and less scary) than I initially thought.

As for progress: with your help, I now have a working version of Alien::premake, and the Build::Premake plugin. Any comments are more than welcome!

I did run into some problems using the 'cd' command, because it complained that the directory didn't exit. Oddly enough, if I used the exact same path, at the exact same point in the script, with File::chdr, it worked fine. So for now I still use File::chdr.

I still have to figure out the calls to requires, and what needs to go in each. And I'm still not doing anything with DESTDIR, but I've managed to use them locally, and they work, which I think is a good first step.

Thank you!

[plicease]

Both are looking pretty good so far. The main thing missing are some tests. Test::Alien::Build is handy for testing plugins (AB has some examples for the core plugins) and Test::Alien can be used for testing aliens (Alien::gmake and Alien::cmake3 are probably two tool-oriented Aliens that will be similar in the testing department).

these requires should be configure requires, since you are useing them in the alienfile you will need to have them installed even if you are doing a system install: https://github.com/jjatria/Alien-premake/blob/master/alienfile#L14

configure { 
  requires 'File::chdir';
  requires 'Path::Tiny';
};

for now you get these when you install AB anyway, but it is good you are declaring them, because we could in theory change the internal implementation to use something different.

This add requires looks right:

https://github.com/jjatria/Alien-Build-Plugin-Build-Premake/blob/master/lib/Alien/Build/Plugin/Build/Premake.pm#L26

not sure what else you might need?

[jjatria]

Thank you for all the help! I've added some tests (which I mostly stole from the projects you mentioned) and the tests are currently passing in Travis. I've also added the documentation, which was missing.

So I think this is done! If you care to take one final look, I can release them to CPAN and close this feature request.

[plicease]

I will try to take a look today!

[plicease]

Finally got to this, sorry last week got away from me. I think this is ready. I have a few notes,

• Should the alien be called Alien::premake5? If premake4 (etc.) projects are built reliably with premake5 probably not worth it.

• Glancing at the premake documentation it looks like only gmake may be supported. Use the Build::Make plugin to force gmake on platforms that default to bsd make if you start seeing failures on *BSD make that have to do with make syntax (it may be worth trying it out on FreeBSD if you have a VM). ($meta->apply_plugin can be used from init to apply a subplugin).

• You can test system install on travis with this:

env:
  matrix:
    - ALIEN_INSTALL_TYPE=system
    - ALIEN_INSTALL_TYPE=share

If you want; you will need to download binary from the premake website, since obviously the travis vms have a much older version of premake. (Alien::cmake3 does this since the version of cmake is too old on travis).

• Alien::cmake3 also downloads binaries for share installs for platforms for which binaries are provided avoiding the time consumed building cmake. Something to consider for Alien::premake. The build time for premake didn't seem as onerous as cmake to me. But something to consider.

I think what you have is great though. Feel free to ignore any or all of these notes. I am excited to see this on CPAN.

[jjatria]

No problem! I figured it was probably something like that.

Should the alien be called Alien::premake5?

Good point. I actually thought about this too, but I don't really know much about premake. I opened a ticket about it on the premake repository, but I suspect they'll say that premake5 is a different beast from premake4. So maybe Alien::premake5 is a better name. Let's see what they have to say, if anything.

Use the Build::Make plugin to force gmake on platforms that default to bsd make if you start seeing failures on *BSD make that have to do with make syntax

I'll keep that in mind. I haven't tested this on anything that is not Linux, so I assume there will be plenty of issues like this.

You can test system install on travis

That sounds like a good idea. I don't think I've been testing system installs with any of this.

Alien::cmake3 also downloads binaries for share installs for platforms for which binaries are provided

I thought about that too, but I temporarily moved away from it while struggling with the version sorting. Will revisit!

I think what you have is great though. Feel free to ignore any or all of these notes. I am excited to see this on CPAN.

Thanks! :D

I've been using it locally to try and make some progress on Alien::Box2D, and my fork seems to be working (although it is currently downloading the sources from a pinned commit from master, which is totally not ideal).

I did run into some interesting questions working on that as well, but they are out of scope for this issue, I guess. In particular, I'm doing the entire installation by hand, including the creation of box2d.pc, which I imagine is not the best way to handle things.

Comments on that are absolutely welcome as well, if you have the time to spare. :)

[jjatria]

Let's see what they have to say, if anything.

And so they've spoken. Apparently, premake4 projects are not guaranteed to work with premake5, so I'll rename the project.

[jjatria]

Here's a question:

Although Alien::cmake3 also has the version name as part of the executable, the Build::CMake plugin (which uses Alien::cmake3) does not. And the %{cmake} helper provided by that plugin inherits from Alien::cmake3, but is not %{cmake3}.

Should I follow this example and provide Alien::premake5, while keeping the build plugin and its helper with the current name?

[plicease]

I would go with %{premake5} since that is the name of the exe. They have designed it easy to keep multiple versions installed, unlike cmake. %{premake} is also probably fine because I don't think you are ever likely to use two different versions in an alienfile. The rationale for Alien::cmake3 having a version number was for it to not conflict with Alien::CMake on case-insensitive systems. Seemed like a good excuse to not support versions prior to 3.0. It will be interesting to see what happens if they ever bump the major version to 4.

[jjatria]

And they are out! Once again, thank you so much for all the help on this. This can now be closed.

Just as an update, I decided to set the name of the distributions as Alien::premake5 and Alien::Build::Plugin::Build::Premake5. The first one defines %{premake5} as the executable. The second one replaces %{premake5} to add the options, and adds %{premake} as a synonym.

Any future comments are always welcome!

(By the way, I sent a forced push to the master branch of the build plugin to replace a botched commit).

[plicease]

cool. Happy to help. Welcome to the AB ecosystem :)