A-B future

[mohawk2]

@devel-chm raised these excellent points on this other issue (https://github.com/Perl5-Alien/Perl5-Alien-Documents/issues/4#issuecomment-53967860):

I do think that some structural changes will need to optimize AB for Alien2 capabilities however, the fact that it is a build-on framework means that we can make the changes and fairly simply port existing AB users to A2B. For most purposes, I hope that the "fixed" API still starts from the same beginnings. It just works a lot better making things work cross-platform.

Here are some of the pieces we have (already!) to work with. This is a list of my thoughts which may differ from others ideas along the same lines:

(1) Use PkgConfig to standardize communication of platform installation information and capabilities. NOTE: we will need to deal with issues of whether and where the pkg-config files would be but it makes sense to me that the vocabulary should still be the same whether or not a distribution pkg-config is present.

(2) AB is improving incrementally but I think a simple refactoring to make the operation symmetric across platforms would really help

(3) I think we should think about using EUMM as the dominant build process for AB (vs MB) for a number of reasons. EUMM is the defacto build process for perl and is supported across ALL perl platforms. Many/all external dependencies actually need the usual make/gcc/... environment to build so a AB module would need a make, a shell, and a compile environment to do more than detect. We could support a set of MB tools or modules to enable platform users to obtain a make+shell+cc build environment and then go straight make from there. Personally, I think the pure perl build tool is the most elegant solution but MB seems to be losing momentum---in part because it is a thankless job to implement EUMM and make in perl. Lots of more enjoyable and interesting things to program out there!

(4) PDL has a large number of external dependencies that could be used as use cases with which to develop this functionality along with a number of motivated developers.

[mohawk2]

I support the use of EUMM.

I would caution that PkgConfig is idiomatic(ish) for Unix, but it is not for e.g. Windows, which uses the registry for that sort of thing. I don't yet have a good answer for that one, but I am thinking about it as that is the main thing stopping a Windows port of Gimp-Perl. It would be nice if we can figure out a more general portable solution. Clearly for Gimp-Perl, there needs to be an Alien::Gimp which can provide relevant information, in a cross-platform way.

[jberger]

1) There are pure-perl PkgConfig implementations, including one in AB already (the one on CPAN had languished, but IIRC has been worked on lately).

2) Not sure I follow, where is the symmetry problem? There is no platform-specific code in AB as yet (though a few choices were motivated by a single platform (I'm looking at you Mac/Mach))

3) eeeeeeeee, I see why this is a consideration, but I have no Make nor MakeMaker foo. AB was able to be built rapidly using MB because it is simply Perl (perhaps a little hideous, but still Perl).

4) Yes PDL can be a great driver of AB, I hope it helps there

[plicease]

I can't say that I agree with using EUMM, MB is being removed from the core, but it works everywhere that matters, and the only reason that EUMM is still in the core is there are core dependencies on it. If they could they'd remove that too.

[mohawk2]

I may be biased as I just did a bunch of stuff on EUMM, but my understanding of the feeling in the Perl toolchain folks is they really dislike MB because they feel it's awful to use - that's why it's being removed. Personally I intend in time to achieve a unification of the two (in terms of interface - implementation, we'll see).

[jberger]

@mohawk2 the guts are pretty bad, but it's not intractable. The "meta" layer (if you would call it that), meaning the bit that lets you add attribute accessors is especially bad. That said, IMO its still a flexibility improvement over EUMM for those of use for whom Make is not in our lexicon.

[mohawk2]

Make can be pretty grody (believe me, I know - I'm fixing dmake). But using EUMM, from a Makefile.PL, that's supposed to completely hide make from the user. You don't find it does that?

[jberger]

I got in as far as some MY:: mess (lacking crucial documentation imo) and got lost, never to return. I probably haven't given it a fair shake. That said, in the case of AB, I can see how eumm is helpful for make-based projects, however, two things.

1) make is not used in all projects that we might be wrapping with AB 2) even if the project does use make, likely it also uses autoconf which requires a bash-like interpreter, which so far, native windows perl installations do not provide

These limit the utility of moving to EUMM (not to zero admittedly). I do think personally that MB is a better solution.

[plicease]

I have had to use postamble to modify the generated Makefile. Maybe in those cases there was a better way? I am not sure. For me personally I've been able to debug Build.PL issues with perl -d in the past.

[plicease]

BTW- Alien::MSYS provides enough of the bash environment to enabling building autoconf style packages on Windows. This is what I use on Windows for Alien::Libarchive. Currently it probably only works reliably if you your Perl was built with gcc, but I managed to get FFI::Raw to build libffi with Alien::MSYS with a Visual C++ build.

[plicease]

If AB can detect the use of autoconf on MSWin32 it would be great if it could inject a dependency on Alien::MSYS and wrap the calls to configure, make, etc.

[jberger]

For me personally I've been able to debug Build.PL issues with perl -d in the past.

This is kinda the essence for me, we as CPAN (and indeed Alien::) authors are good at Perl and I know how to debug it. This is why I like sticking with a pure perl toolchain as far as possible.

Alien::MSYS provides enough of the bash environment to enabling building autoconf style packages on Windows.

Now that is good news! I wouldn't mind figuring out some way to optionally depend on it!

[jberger]

If AB can detect the use of autoconf on MSWin32 it would be great if it could inject a dependency on Alien::MSYS and wrap the calls to configure, make, etc.

Open a feature request!

[plicease]

If AB can detect the use of autoconf on MSWin32 it would be great if it could inject a dependency on Alien::MSYS and wrap the calls to configure, make, etc.

Open a feature request!

I can probably do one better than that I can do a PR :)

[devel-chm]

Hi, great discussion!

The rationale to go with EUMM was strictly pragmatic and KISS (keep it simple stupid) based: using EUMM is very simple in perl, is supported by core perl, is widely available.

In sum, I think it would be simple to to EUMM+quick fail followed by fixes from MB and the perl level (and as someone who doesn't grok MB it definitely seems the simpler way to go).

As far as I know, make can be used for any project by using the trivial wrapper:

all: Do package install process from shell

Actually, most pure opensource packages seem to be using dmake or else some wrap up of a unix environment like MSYS/MinGW or cygwin.

The use case for comparison here has to be an Alien module implementation and not the package itself since this is all about the AB layer.

--Chris

On Sat, Aug 30, 2014 at 5:31 PM, Joel Berger notifications@github.com wrote:

I got in as far as some MY:: mess (lacking crucial documentation imo) and got lost, never to return. I probably haven't given it a fair shake. That said, in the case of AB, I can see how eumm is helpful for make-based projects, however, two things.

1) make is not used in all projects that we might be wrapping with AB 2) even if the project does use make, likely it also uses autoconf which requires a bash-like interpreter, which so far, native windows perl installations do not provide

These limit the utility of moving to EUMM (not to zero admittedly). I do think personally that MB is a better solution.

— Reply to this email directly or view it on GitHub https://github.com/Perl5-Alien/Alien-Base/issues/43#issuecomment-53971101 .

[jberger]

The use case for comparison here has to be an Alien module implementation and not the package itself since this is all about the AB layer.

@devel-chm I'm not sure what you mean here, AB provides a subclass of MB that offloads all of the work of building the library to the distribution builder. This means that the two are inextricably linked together. The MB subclass (Alien::Base::ModuleBuild) has hooks at nearly every phase allowing the installer to configure, build test and "install" in phase with the Alien:: module itself.

[devel-chm]

I mean the Alien module should use EUMM. Not sure what the implications for AB would be.

On Sat, Aug 30, 2014 at 5:55 PM, Joel Berger notifications@github.com wrote:

The use case for comparison here has to be an Alien module implementation and not the package itself since this is

all about the AB layer.

@devel-chm https://github.com/devel-chm I'm not sure what you mean here, AB provides a subclass of MB that offloads all of the work of building the library to the distribution builder. This means that the two are inextricably linked together. The MB subclass (Alien::Base::ModuleBuild) has hooks at nearly every phase allowing the installer to configure, build test and "install" in phase with the Alien:: module itself.

— Reply to this email directly or view it on GitHub https://github.com/Perl5-Alien/Alien-Base/issues/43#issuecomment-53971642 .

[jberger]

As of right now, the Alien::Foo module (which the author uses to provide libfoo) is not dependent on EUMM or MB directly, it is dependent on Alien::Base::ModuleBuild, which wraps the entire library build process. I hope you understand this bit already. What you are suggesting is moving all of that functionality to a subclass (or else helper functions?) for EUMM. I'm not sure why.

[devel-chm]

Just mentioning my thoughts. If I were sure I had the correct approach I'd be coding already. :-)

Similarly, if using MB makes sense, then that is fine too but my experience is MB makes for klunky compiles and long build times but that isn't a showstopper if the rest works well.

--Chris

On Sat, Aug 30, 2014 at 6:15 PM, Joel Berger notifications@github.com wrote:

As of right now, the Alien::Foo module (which the author uses to provide libfoo) is not dependent on EUMM or MB directly, it is dependent on Alien::Base::ModuleBuild, which wraps the entire library build process. I hope you understand this bit already. What you are suggesting is moving all of that functionality to a subclass (or else helper functions?) for EUMM. I'm not sure why.

— Reply to this email directly or view it on GitHub https://github.com/Perl5-Alien/Alien-Base/issues/43#issuecomment-53972108 .

[plicease]

I was working on decoupling the installer from the Module::Builld class in Alien::Libarchive. In fact they are separate distributions on CPAN so that libarchive can be used as either an optional (use if available) or specific (eg. version > 1.2) prerequisite via Alien::Libarchive::Installer, or as a general, I don't care so long as it works prerequisite via Alien::Libarchive. It occurs to me that if we adopted a similar strategy in AB, that it would be easier to support both EUMM and MB.

I'd like to keep the MB base class as independent of any EUMM and not as a comparability mode. So long as there is at least one developer with time to support the MB version this seems reasonable.

Any EUMM alternative would require at least one core developer willing to support it.

Even without a EUMM implementation I think decoupling the installer from the MB class is worthwhile.

[plicease]

@jberger any possibility we could get a dev version out to CPAN? I'd like to see some of the features and bug fixes we have made tested by cpantesters. I think a prod version could follow fairly shortly if things look good. I'd be happy to do the actual upload if you don't have the time.

[mohawk2]

I'm very keen to see a dev release too. I note that I don't have PAUSE permission for Alien::Base[*]. @jberger, how do you feel about adding @plicease and/or me and/or @zmughal as co-maints on PAUSE?

[jberger]

I'll admit i was keeping that as a trump card in case this whole committee approach failed, but i have been been very impressed by the progress that has been made in very short order. Have any patches been made to the Acme modules? I have typically pushed all three at the same time.

[jberger]

OMG I haven't actually logged into PAUSE in so long! Where is anything?

[mohawk2]

https://pause.perl.org/pause/authenquery?ACTION=share_perms (on left menu, "Change permissions")

[jberger]

@mohawk2 you know, I don't know your pause ID

[jberger]

I have added @plicease and @zmughal

[jberger]

added @devel-chm

[mohawk2]

ETJ.

[plicease]

There is a PR for Alien-Base-Extras, which brings it up to speed with regard to Alien-Base. Two things should probably be added:

1. Acme-Alien-FordPrefect should use alien_isoloate_dynamic
2. It should also use --with-pic #47

I'd like to resolve #47 and then I think we are ready for a dev release.

[mohawk2]

Aye - let's press ahead!

[plicease]

Okay here are the changes that I think are needed for Alien-Base-Extras:

https://github.com/Perl5-Alien/Alien-Base-Extras/pull/2

They can wait for the next production release of Alien::Base. Smokers aren't going to be able to test dev AB with the Acme modules anyway.

[jberger]

added @mohawk2

[plicease]

I sent 0.004_01 to CPAN

https://metacpan.org/pod/release/PLICEASE/Alien-Base-0.004_01/lib/Alien/Base.pm

(fingers crossed)

[mohawk2]

I want to be extremely clear here, but I don't understand - why are the two other modules external to AB? Surely they effectively constitute test-input data for AB? Shouldn't they be (re-)incorporated? If there are pressing reasons why not, could someone guide me?

[plicease]

They are separate distributions, I don't think they belong in the AB tarball. They test the interaction between AB and an Alien dist. My tendency would be to go there other way and actually make them three separate repositories, but I don't personally see a strong need to change it either way.

[plicease]

Archive::Extract is removed from the core as of Perl 5.20:

iscah% corelist Archive::Extract

Data for 2014-08-20
Archive::Extract was first released with perl v5.9.5, deprecated (will be CPAN-only) in v5.17.9 and removed from v5.19.0

Most packages are bundled as .tar.gz, for which Archive::Tar would be sufficient (still in the core). Archive::Zip or Archive::Extract could be injected as a prerequisite for Alien dists that do require less common archive formats. At the moment there isn't usually a way to determine from the Module::Build properties which archive formats support will be required.

There is also Archive::Extract::Libarchive and Archive::Libarchive::XS (and ::FFI and ::Any) which don't need a filename extension to extract from an archive.

[devel-chm]

The progress here has been amazing (as in I haven't even had time to look at the code changes!). I recommend staying in CPAN developer release status for a while until any changes in API or spec stabilize.

Also, regarding releases, having a consistent release manager helps for finding things but you might consider allowing the PAUSE co-maints to push a developers release as desired but have jb do the "official stable" ones.

--Chris

On Thu, Sep 4, 2014 at 4:40 PM, Graham Ollis notifications@github.com wrote:

Archive::Extract is removed from the core as of Perl 5.20:

iscah% corelist Archive::Extract

Data for 2014-08-20 Archive::Extract was first released with perl v5.9.5, deprecated (will be CPAN-only) in v5.17.9 and removed from v5.19.0

Most packages are bundled as .tar.gz, for which Archive::Tar would be sufficient (still in the core). Archive::Zip or Archive::Extract could be injected as a prerequisite for Alien dists that do require less common archive formats. At the moment there isn't usually a way to determine from the Module::Build properties which archive formats support will be required.

There is also Archive::Extract::Libarchive and Archive::Libarchive::XS (and ::FFI and ::Any) which don't need a filename extension to extract from an archive.

— Reply to this email directly or view it on GitHub https://github.com/Perl5-Alien/Alien-Base/issues/43#issuecomment-54541337 .

[jberger]

I want to be extremely clear here, but I don't understand - why are the two other modules external to AB?

Do you mean the Acme modules? They have to be in separate releases. For quite a while I had some "simulation" testing but it just wasn't accurate enough. As for the question of should they be in the same github repo, I tend to agree with @plicease that if any change was made they would be in three separate repos, but I don't have strong opinions on that.

To archive modules, I tend to think we should just lean on the cpan version of Archive::Extract for now, but alternatives could be considered.

[jberger]

This was the commit that removed t/zz-example.t that actually used the dontpanic modules as a test: f0d1d81a828f910b321ae5528062573af2a40cb5

This was the commit that removed the dontpanic files all together: a374bafb0dfb498fbf4c7dbd29ca8a22fb5d115b

[plicease]

To archive modules, I tend to think we should just lean on the cpan version of Archive::Extract for now, but alternatives could be considered.

It isn't a priority but I wanted to raise it.

Also with Archive::Extract I don't think there is any guarantee that anything other that tar format will work, since it uses either the command line or Archive::Zip all optionally.

[plicease]

I am preparing for another dev release, unless there are any objections.

[mohawk2]

I say permanently "release early, release often".

[plicease]

0.004_02 is on its way :)

[plicease]

https://metacpan.org/pod/release/PLICEASE/Alien-Base-0.004_02/lib/Alien/Base.pm

[plicease]

Anyone have an OS X they want to test on:

http://217.199.168.174/cgi-bin/cpantestersmatrix.pl?dist=Alien-Base

Everything else is looking good so far.

[jberger]

And this would drive me nuts too. I would get every weird bsd variant and no Mac. I would have to start begging around the interwebs

[plicease]

Don't you care about MidnightBSD? It's the new DragonFlyBSD!

[plicease]

I created some rpms in OBS for Alien::Base, Acme::Alien::DontPanic and Acme::Ford::Prefectwhile I was working on #39 and #65. For those that aren't aware OBS is a automated build system for rpm packages. When you commit changes in OBS it automatically builds not just that package, but any packages that depend on it. It might therefore be a good tool to add to our automated testing process. Synchronizing git and OBS commits may be the tricky bit.

[plicease]

I also intend on creating an Acme::Ford::Prefect::FFI or something similar to test the interaction with Alien::Base and FFI::Raw, which is an interest of mine. Just a heads up.

[mohawk2]

I am very keen to see support for FFI - as mentioned before, I want to make Inline be the way FFI is done.