Overhaul GNU Autotools usage

I first noticed originally packaging this for Arch that the Autotools stuff was a bit wonky but let it go and moved on. With v4.4.0 breakage (#335) I had a closer look and there are several things that are a bit ... sideways. The version stuff isn't handled right, the VCS tracked bits don't work properly between releases, the tooling like make distcheck that should catch packaging issues early just doesn't work, there are some shell scripts that echo some of their commands but not others making debugging a bit of a goose chase, and likely other bugaboos.

I don't expect everybody to be an Autotools expert (and only much pain has made me even half-way competent with it). This issue is actually specifically so if the author would like to see these issues fixed they can assign the issue to me and I'll get to it sometime in my free time. No promises about when that is, just that it will be on my list.

Another concern with autoreconf is the autotools installed version "dependence".

I've had such issues with other projects in the past, when locally regenerating the necessary files did not work to get the ball rolling.

I ran into a similar problem just a few days ago, when using autoconf 2.71 on my dev machine and autoconf 2.69 on RPi 3 (the latest version available for that OS) see my post above. The 2.69 generated configure worked so I was happy about that, but then to my surprise configure did not work properly when generated with autoconf 2.71 and failed to detect NEON with exactly the same configure.ac source. Go figure.

Another concern with autoreconf is the autotools

You lack an understanding of the fundamental difference between the two terms

"tarball" (aka "source distribution") and
"snapshot" (aka "git clone" or "git archive").

They have distinct meaning, and it is a grave mistake to confuse them with each other, and the source of your colorful autotools habits. git is not a software distribution tool, but a development collaboration tool.

autoconf 2.71 works fine and distchecks the Mergesium autotools branch (aka #370) without errors. tarballs generated with autotools 2.71 build correctly both when targetting the build host, and when cross compiling. Cross compiling for Win32 using the MingW32 toolchain (--host=x86_64-w64-mingw32 CXX=x86_64-w64-mingw32-g++-posix) works correctly without without intervention beyond the "-posix" bit. Likewise cross compiling for --host=aarch64-linux-gnu. make install fails for cross-compiled builds because the completions are generated by running the compiled binary, which cannot be done with cross-compiled binaries. That is a problem separate from 2.71 vs other autotools versions.

@alerque when you say 'distcheck works' you mean 'in the #370' branch, right ❓ I believe when @genivia-inc says 'doesn't work' he means the master branch.

The two of you are not on the same wavelength at all. I believe I am on @alerque's wavelength, as I find I understand the subtleties of his comments that @genivia-inc clearly overlooks. I don't yet agree with @alerque suggestion that packages should depend on bash-completions but that is a separate topic, above that I've asked about, and have not yet received a response. Other than that issue, I'm am in complete agreement with his advice.

@drok For the record a "tarball" is not defined as a "source distribution". It is a means to distribute source in a convenient format with tar and gz. Look up the definition. No need to get sarcastic in your posts.

What you want to put in (your) archive or want to package is up to you, to keep or delete things. But PR #370 and suggestions to delete files applies to all of this repo, right? I just want to be clear about this that it is my decision to keep or delete files from the repo in a way to release software while avoiding potential problems, such as configure errors. When folks download the ugrep-master.zip from the repo or use git clone or any other way then they get the complete and buildable set of files. If you are referring to your archive or your way of packaging for distribution, then you need to communicate that important differentiating aspect in the narrative. Otherwise, this creates confusion and we keep going in circles.

Please read my autoconf 2.71 versus 2.69 comment. Which is that autoconf 2.69 behaves differently for this specific use of AC_PREPROC_IFELSE when 2.71 does not. There is no logged info that tells me why. Requiring autoconf or autoreconf can lead to such unexpected issues for sure. Sadly, when this happens it will fly under the radar because when the build completes it will lack SIMD optimizations when SIMD optimizations are actually possible. So you should check the configure output and compiler statements to verify that SIMD (SSE2/AVX2/AVX512/NEON) are actually used appropriately, to verify that the configure.ac is acceptable.

The updated configure.ac I've committed is in a pretty good shape IMO. It works fine on several platforms I've tested. I've also tested cross-compilation to some extent. It doesn't misdetect and skip SIMD optimizations.

I could not reliably get the PR #370 autotool and makefiles to work. If it compiles, then it leaves out SIMD optimizations or applies them incorrectly with ax_ext.m4. There is also no such thing as runtime NEON detection in a way that will be useful with ugrep. It's compiled with NEON, or it's not.

EDIT: I want to be a bit more specific here: I've tested SIMD detection accuracy of my updated configure and makefiles on Debian x64, Ubuntu x64, MacOS M1 (AArch64), MacOS x64, RPi 3 (NEON), RPi zero (no NEON), Android termux (AArch64), Cygwin (x64). By contrast, the suggested ax_ext.m4 script fails to detect NEON on RPi 3 and I noticed problems with AVX (see my suggested fixes). Furthermore, ax_ext.m4 uses CPUID, which is fine when not cross compiling. But for cross compiling the host machine properties should be used, which I've done in my updated configure scripts.

Read this para by @alerque .

It isn't just those files, if you git clone or download the git archive generated snapshots you don't gave a configure script either. That and all the other files we're mentioning are part of the distribution tooling that is either expected to be used as generated without messing with it (as in using the make dist generated source tarballs) or generated on the build system using autotools.

On one hand,

He refers to git repo content as "git clone", "git archive", "snapshots", which don't "gave" (sp! - I think he meant "have") a configure script.

On the other hand,

He also refers to "distribution", "source", "tarballs", which is something generated by the make dist command, and which contains "distribution tooling".

He emphatically highlights that these two ideas are distinct, mutually exclusive, and he calls attention specifically to the contrast between them. The purpose of the paragraph is to point out the mutual exclusivity of the two ideas, and to state precisely what makes them distinct (the "configure" script, and other mentioned files which collectively are known as "distribution tooling"). He says that the "distribution tooling", which include the "configure" script do not exist in the "git clone" or "git archive" or "snapshot" representation of a project, but are generated by, and included, embedded, appended, or provided by the "make dist" command when the "generated source tarball" is created.

The language is very clear and verbose, even redundantly so.

This verbosity and redundancy would not be necessary when talking with another person who is familiar with the autotools process. The described concepts are elementary to someone who has RTFM. However, he is addressing you, and you have demonstrated amply that you have not RTFM. In his paragraph, @alerque is attempting to politely teach you elementary concepts of autotools that you ought to know given the experience you profess, while you demonstrate at every utterance to be entirely unfamiliar with.

Your response about "SIMD" or "ax_ext.m4" refer to irrelevant minutiae. They are not unimportant, but they are irrelevant until you grasp the basics of the framework, like what the typical workflow is, what the steps of the workflow are, and what the role of various files involved at each step is. The fact that your response refers to such detail while glossing over the big elephant shows you have not seen the big elephant yet.

We are going in circles for two fucking weeks around a half-a day problem because we're not only solving a problem, but also teaching you how and why the solution works, while you have clearly made zero effort to even become informed of the autotools functionality as intended and designed by its creators. You consistently insist on using autotools according to your preferences.

Cleaning up the mess is a half-a-day effort. Writing the configure.ac and Makefile.am needed to compile your C++ project from scratch would take less than that (assuming proficient working knowledge of autotools). I have taken two days to produce the PR that cleans up the mess. But for some reason, two weeks later, while you have a clean, working implementation served to you on a platter, you still struggle with the basics of where the "configure" file comes from. Do you not wonder why it's taking so long?

The likelyhood of you having found a breaking bug with the basic functionality of autotools 2.71 is vanishingly, vanishingly (repeated for emphasis) small. 2.71 was released with the known issue that the configure.ac syntax has been made stricter, and that some existing configure.ac scripts that worked (accidentally) with 2.69 are broken with 2.71 and need to be fixed.

That said, @alerque specifically gave you this advice (emphasis is mine):

target everything in configure.ac to whatever version of autoconf you'll be using on whatever system is going to be running make dist (whether that is local or CI) and don't worry about what versions might be on target systems.

You ignored this, probably because you did not understand it; it is important advice as it goes to the heart of your struggles. I will attempt to unpack it.

Unpacking this advice verbosely: you need not worry about what version of autoconf might be installed on a user's system (ie, 2.71 for greater certainty). When you distribute a source tarball, your C code is built with scripts of your choice (eg, generated by the autoconf you want to use - the local system in @alerque's sentence, or whichever is installed on the CI system, refering to Github's ubuntu-latest which is CI OS you chose). Your configure.ac doesn't need to be compatible with any other autoconf versions than the one you choose. This is like choosing -std=c++11 instead of some other standard. It's your choice, and don't need to worry about the compiler supporting other standards. @alerque was more succinct than my unpacking, but you completely glossed over it, and continued to complain about "autoconf 2.71 being broken", after he told you it would not matter if 2.71, 2.72 or 2.79 are broken. If you choose 2.69 or something else, that is what the source will be compiled with, once you understand and follow the standard autotools workflow, instead of your preferred hacks. Future versions are not relevant, until, and if you decide to migrate to another version. You ought to give some thought to migration, given it's unnecessary, and probably choose to not migrate for as long as it is an option. That said, migration would only be an issue if your configure.ac is flimsy, and yours is.

In short, it would not matter if autoconf 2.71 had a bug or many bugs. If you follow autoconf documentation, you would not be affected by any of the bugs that autoconf 2.71 may or may not have.

I'm watching you fumbling with all this with the amusement of someone watching a monkey attempting to use a typewriter. I don't have the patience to wait for a Shakespeare play to emerge from your output tray.

I mean no disrespect or sarcasm, but I do mean to convey the frustration of trying to help you "get" how autotools works. I honestly regret getting involved in your project, and this is where I leave. I have used every ounce of patience I have (not being a naturally patient, politically correct person), and have done everything in my power to politely guide your adventure, but I have nothing left to give.

However, I will continue to maintain the Mergesium fork of your program, for the purpose of providing clean, portable autotools tooling for your program.

I have written this because your earlier comment that you believe to be surrounded by "smart" (your sarcastic quotes) people who are usually wrong. I now have a better understanding of your definition of "wrong", and felt it necessary to point out that you are full of shit, superficial and arrogant. I am happy to be "wrong" according to you, but be done with the nightmare of interacting with you.

Closing for lack of respect by some of the participating parties. No need for ad-hominem attacks on my person when I point out some problems I found, for which I have sufficient evidence gathered with testing. I'll be happy to provide more evidence with screen shots. These attacks and derogatory comments violate our code of conduct.

In the meantime, v5.1.2 is released with updated configure and makefiles, some of which is based on suggestions shared. Many thanks for those, at least.

Genivia / ugrep

Overhaul GNU Autotools usage #339