Autotools generated scripts are under version control

[ryao]

I am told that projects only generate these scripts when tagging releases, but we seem to have them under version control. Why is that?

[behlendorf]

They're right. Typically, these files aren't tracked under version control and are just included in the tagged tarballs.

However, since I've yet to tag a stable release and we have lots of people still pulling from the git repo directly it was convenient to include these autotools products. Otherwise you might need to find a system with the right version of autotools installed to generate the configure script, so you could then build the project. The products of autotools are portable, but things like the configure.ac files and Makefiles.am. They are often tied more closely to specific versions of autotools than people would like to admit.

That said, perhaps we getting to the point now where most people shouldn't be pulling directly from the master branch on Github. They can use one of the tagged -rc tarballs or either the versions integrated with the downstream distributions. If that's now more than norm we could seriously consider removing these files from the repo.

[chrisrd]

For what it's worth, I'd vote to keep the generated scripts in git for (my) convenience.

[behlendorf]

Well certainly keep them for a while yet, but perhaps we can drop them shortly before tagging a stable 0.6.0 which I think it still a ways off.

[ryao]

@chrisrd Which distribution do you use?

If I provided a way for you to compile a kernel, an initramfs image, SPL and ZFS on your distribution using Gentoo's package manager (e.g. inside /opt/gentoo), would you use it?

[chrisrd]

Debian. No (but thanks!). I have my own workflow in place to manage the bits and pieces of kernel-land I'm interested in whilst they're in flux. But I'm looking forward to the day when I'm no longer on the bleeding edge and can simply use standard packages!

[behlendorf]

My current intention is to drop the generated autotools script for the repositories once the following RPM DKMS changes have been merged in to an official tag.

This should allow us to start posting just tarballs and source RPM packages to the website for general use. I'll of course also keep prominent links to both the Ubuntu/Debian PPA and Gentoo packages. Building directly from the repositories can be demoted to a developer only thing where they should be expected to have the autotools installed.

zfsonlinux/spl#136 zfsonlinux/zfs#830

It still may take a while to get these merged but I wanted to give everyone a heads up. Does this seem reasonable to everyone, are there any concerns?

[ryao]

This will cause no problems in Gentoo. The package manager will generate the autotools scripts as part of the build process when building directly from ZFSOnLinux GIT.

[Rudd-O]

Having the autotools scripts in the repo (makefile.in and the like) isn't a "convenience". It's a huge mess. It makes merging, rebasing and other repo operations (which should be straightforward) into a NIGHTMARE, because every single autotools out there generates different scripts (which are just a mess of shellscripting) which makes it impossible to fish out the relevant changes for merging.

I vote for them to be removed since they have caused me indescribable pain and completely demotivated me from contributing more.

In fact, I would fully support ditching autotools altogether and using something more rational like waf, which doesn't require at all littering the repository with meta-meta-meta-meta-meta files.

[ryao]

@Rudd-O Do what I do. Extend your system package manager to pull from a branch on github, run autotools, build and install. Then Ignore the autotools scripts when doing git operation. Next, push to the designated branch on github when you need to test a change and finally, let your package manager handle autotools.

I find that to be the only sane way of dealing with the autotools scripts.

[behlendorf]

More convenient for users, not for developers. By this point I hope most users are sticking to the tarballs or using whatever solution is best packaged for their distribution.

Anyway, once we get the RPM DKMS changes in place I'm happy to drop the autogen.sh products. Just be patient with me for a few more weeks.

[Rudd-O]

@ryao: my scripts already do autogen.sh (well, they used to, when I wasn't using DKMS -- I am now) because that was the only sane way to operate. The problem is that the garbage autotools-generated scripts make it nigh impossible to merge branches. My process today consists on reverting my own autotools-script changes, generating a diff between the upstream autotools scripts and mine, applying that diff as a commit, then doing the merge, then backing out that autotools-scripts-diff out using rebase --interactive. It's crazy enough that I fear it will give me trichotilomania.

Would you guys entertain patches of the build sys to a full port to waf? No more garbage scripts!

[behlendorf]

Would you guys entertain patches of the build sys to a full port to waf? No more garbage scripts!

No. Autotools may have it flaws but it's still the most flexible and portable thing for a complicated project out there.

[Rudd-O]

I respectfully beg to differ with your judgment in this case.

waf is much more portable (it's fully self-contained, works everywhere, doesn't require users or developers to install anything, no autoconf, no autoheader, no automake, just to extend the build system), it's way more flexible (for example, it supports multiple configurations like DKMS or no-DKMS out of the box, unlike the autoconf hacks we have had to use at make time), its configuration is WAY easier to understand and extend than the arcane autotools files (m4, really?), and waf doesn't litter the project at all with random files.

Oh, and it's much faster too. Not only does waf build everything in parallel, its configure tests (the lengthiest part) are much, much faster than configure, and it only rebuilds what's necessary since waf understands the whole dependency chain. For me, this is the biggest, by far, the biggest win. If anything else, you should expect build and compile times to be slashed in half (or more for multicore systems).

The SAMBA project successfully replaced autocancer with waf (and they have vastly more complicated portability requirements than us). I personally used waf to replace an ant build system at Cloud.com, slashing compile times from 15minutes+ to less than 90 seconds.

The hard part is porting the M4 tests, really, and I would be doing that part.

There are real benefits to using something like waf. I respectfully suggest you reconsider your initial assessment.

[ryao]

@Rudd-O Why are you having problems merging branches due to the autotools scripts? If the only changes to the files that ./autogen.sh touches are being made at upstream until you try building the software, then there should be no issues.

I do not think that anyone will commit to switch from autotools before seeing it in action. If you want to change people's minds, you will need to rewrite the build system in waf, without any guarantee that your work will be adopted. If you do that, I am sure that @behlendorf would evaluate it.

[Rudd-O]

Yes, absolutely, the plan is for me to rewrite the build system using waf, then be evaluated by you guys, and then be committed if it passes muster. I just didn't want to do the whole endeavor (it is going to be a few days' work) if there was a GOOD reason why the build system should not be migrated to waf.

[ryao]

I spoke to a few Gentoo Python developers about this. They all consider waf to be inferior to autotools. According to them, software using it requires more effort to support and maintain downstream. That is not a solid reason against it, but it is not the response that I hoped to hear.

[Rudd-O]

I suggest to ask users of build systems rather than users of languages that do not need build systems instead. Ask the SAMBA devs.

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Richard Yao reply@reply.github.com wrote:

I spoke to a few Gentoo Python developers about this. They all consider waf to be inferior to autotools. They consider software using it to require more effort to support and maintain downstream. That is not a solid reason against it, but it is not the response that I hoped to hear.

---

Reply to this email directly or view it on GitHub: https://github.com/zfsonlinux/zfs/issues/718#issuecomment-7268048

[ryao]

One of these people maintains the samba ebuild in Gentoo.

[behlendorf]

In theory waf looks very cool, although I strongly suspect it will break down in practice. The devil's always in the details for this sort of thing.

I'm also sure that rewriting the build system isn't a few days of work, it's a few months of work. If you really could replace the entire build system and all its functionality with waf in just a few day that would be a pretty compelling argument.

But even then. It's hard argument to make that we just discard what we have now which works well and replace it with something which may or may not meet all our needs.

[Rudd-O]

We will not replace anything until whatever replacement is in line, actually does everything that the former replacement does, in a better way. So rest assured, I won't be submitting any destructive patches. :-)