build: Non-recursive makefiles

[Explorer09]

Status of this branch

• Binaries can build properly. make distcheck passed.
• The 'flex' binaries have identical size (between the recursive makefiles and non-recursive one) when stripped, but the binaries are not bit-identical. May need to analyze why the builds was not deterministic as I expected.

Description

This converts the build system of flex to use non-recursive makefiles, except for "po" and "tests" directories which are technically too complex to convert. The main advantage of using non-recursive makefiles for Flex is that cross-directory dependencies can be handled better. The reduced total size of generated makefiles is a second advantage.

This is part of my rewrite of the bootstrap mechanism of Flex in order to make it portable to non-GNU "make".

[westes]

The goals of non-recursive make and making flex's build system portable to non-GNU make are (at least logically) separate.

Can you point out where we could change things to make them not GNU-make specific? That seems worthwhile -- to the extent that automake allows this, that is.

Given that po/ is handled by gettext, is there any possibility of incorporating po/ into the non-recursive scheme? I assume the answer is "no".

The tests/ directory is just a mess of files that is only going to grow over time. I'm not sure we would want to incorporate it into a non-recursive scheme even if it were possible. If anything, I'd actually like to split tests/ into subdirectories just to make it much less of a sprawl. (But that's super low priority for me, so I haven't given any thought to how practical or good an idea this might turn out to be.)

Given that we have po/ and tests/ as recursive, do we really want a build system that is sort-of-recursive, depending on which directory we're talking about?

Having to repeatedly list subdirectories as the examples listed in the page you link seems like a huge net loss for no particular gain.

Also, one of the idioms we have now is:

cd doc && make

How does your replacement build system provide for this kind of thing? I'm thinking particularly of the case where the files are edited in a subdirectory and the Makefile is not in the directory any longer. I think this would make running the Makefile from emacs harder with no particular upside.

[Mightyjo]

I don't care for this. It just looks different to be different. Hold off until I answer your argument about make being POSIX. Work got busy.

On Fri, May 10, 2024, 17:33 Will Estes @.***> wrote:

The goals of non-recursive make and making flex's build system portable to non-GNU make are (at least logically) separate.

Can you point out where we could change things to make them not GNU-make specific? That seems worthwhile -- to the extent that automake allows this, that is.

Given that po/ is handled by gettext, is there any possibility of incorporating po/ into the non-recursive scheme? I assume the answer is "no".

The tests/ directory is just a mess of files that is only going to grow over time. I'm not sure we would want to incorporate it into a non-recursive scheme even if it were possible. If anything, I'd actually like to split tests/ into subdirectories just to make it much less of a sprawl. (But that's super low priority for me, so I haven't given any thought to how practical or good an idea this might turn out to be.)

Given that we have po/ and tests/ as recursive, do we really want a build system that is sort-of-recursive, depending on which directory we're talking about?

Having to repeatedly list subdirectories as the examples listed in the page you link seems like a huge net loss for no particular gain.

Also, one of the idioms we have now is:

cd doc && make

How does your replacement build system provide for this kind of thing? I'm thinking particularly of the case where the files are edited in a subdirectory and the Makefile is not in the directory any longer. I think this would make running the Makefile from emacs harder with no particular upside.

— Reply to this email directly, view it on GitHub https://github.com/westes/flex/pull/652#issuecomment-2105297304, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVJXIOAXTM6QTQZO45C7WDZBU4K7AVCNFSM6AAAAABHRHOK46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGI4TOMZQGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[westes]

To condense my questions down a fair bit and ask from a higher level perspective:

What problem does this solve that flex is having?

None of the reasons that the nonrecursive make page gives are relevant. Maybe the multi-threading thing is technically true, but if you're rebuilding flex so often that you notice, I would gently suggest that you're doing something else fundamentally wrong.

[Explorer09]

@westes

What problem does this solve that flex is having?

None of the reasons that the nonrecursive make page gives are relevant. Maybe the multi-threading thing is technically true, but if you're rebuilding flex so often that you notice, I would gently suggest that you're doing something else fundamentally wrong.

The problem I was trying to address is when two 'make' instances running in parallel, and cross-directory 'make' calls exist, then two instances of 'make' might touch the same file at one time, causing contention.

This problem is rare, I admit, but possible.

For flex, the case is one can run make -C doc/ all and make -C src/ all almost at the same time, and because of cross-directory dependency, two 'make' instances might try to build (with compiler and linker) the same flex binary at the same time.

"make -C doc/ flex.1" -> flex.1
                            |
                            |
                            V
"make -C src/ flex" -----> flex (program)

(Unix 'make' is never designed to touch or build files atomically. The usual workaround is to tell users to be careful and not to invoke two 'make' instances that might touch the same file simultaneously.) That's how cross-directory dependency can be tricky. Recursive makefiles handle this problem by ensuring subdirectory makefiles be run only in a specific order, and no two subdirectory makefiles can be invoked at the same time (which limits parallelism).

I wished there is an alternative way to handle this, and make the solution useful in the long run.

And regarding the idiom you mentioned:

cd doc && make

Yes, I am aware of that, and I was reluctant to change tests/Makefile.am for that very reason. But for "doc" directory, this idiom is the problem. Unless everything in the "doc" directory depends on nothing outside the directory, I don't think it is safe to have a makefile to be invoked there. (Just my opinion.)

[Mightyjo]

I understand the issue that you're having with the doc directory, but it comes back to the misunderstanding we were discussing regarding the name of the flex binary at various stages of the bootstrap and build process. Let me take a crack at fixing the libtoolized make files before we go down this path.

On Fri, May 10, 2024, 20:27 Kang-Che Sung (宋岡哲) @.***> wrote:

I'm sorry but I need some sleep before I have the energy to explain all questions...

@westes https://github.com/westes

What problem does this solve that flex is having?

None of the reasons that the nonrecursive make page gives are relevant. Maybe the multi-threading thing is technically true, but if you're rebuilding flex so often that you notice, I would gently suggest that you're doing something else fundamentally wrong.

The problem I was trying to address is when two 'make' instances running in parallel, and cross-directory 'make' calls exist, then two instances of 'make' might touch the same file at one time, causing contention.

This problem is rare, I admit, but possible.

For flex, the case is one can run make -C doc/ all and make -C src/ all almost at the same time, and because of cross-directory dependency, two 'make' instances might try to build (with compiler and linker) the same flex binary at the same time.

"make -C doc/ flex.1" -> flex.1

                        V

"make -C src/ flex" -----> flex (program)

(Unix 'make' is never designed to touch or build files atomically. The usual workaround is to tell users to be careful and not to invoke two 'make' instances that might touch the same file simultaneously.) That's how cross-directory dependency can be tricky. Recursive makefiles handle this problem by ensuring subdirectory makefiles be run only in a specific order, and no two subdirectory makefiles can be invoked at the same time (which limits parallelism).

I wished there is an alternative way to handle this, and make the solution useful in the long run.

And regarding the idiom you mentioned:

cd doc && make

Yes, I am aware of that, and I was reluctant to change tests/Makefile.am for that very reason. But for "doc" directory, this idiom is the problem. Unless everything in the "doc" directory depends on nothing outside the directory, I don't think it is safe to have a makefile to be invoked there. (Just my opinion.)

— Reply to this email directly, view it on GitHub https://github.com/westes/flex/pull/652#issuecomment-2105406796, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVJXILN75LIXREFE6PCXYTZBVQYTAVCNFSM6AAAAABHRHOK46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGQYDMNZZGY . You are receiving this because you commented.Message ID: @.***>

[Explorer09]

@Mightyjo

I understand the issue that you're having with the doc directory, but it comes back to the misunderstanding we were discussing regarding the name of the flex binary at various stages of the bootstrap and build process. Let me take a crack at fixing the libtoolized make files before we go down this path.

Excuse me. What misunderstanding?

(Just to attach the flex bootstrap diagram again for reference:)

[Explorer09]

@westes Just to clarify some points:

1. This is still a draft (GitHub allows me to file draft PRs like this in order to grab some early comments), and I'm not expecting this one be merged very soon.
2. I am aware of the weaknesses of the non-recursive makefiles, and which uses cases are unfit:

(a) Non-recursive makefiles have to be invoked from the top level; people would lose the ability to just rebuild a subdirectory. For subdirectories that are mostly self-contained (that depend on little from the outside), the lack of building a subdirectory is a loss. But, for subdirectories with a lot of inter-directory dependencies, the lack of such ability may be considered a feature, not a bug with the non-recursive approach. (b) Any inference rule (aka. suffix rule) in the subdirectory makefile fragments would affect the whole project's build. There is less freedom to choose file extensions for inference rules for subdirectories as you must now aware of suffix collisions.

The "tests" directory in Flex does use some custom suffix rules, so the weakness (b) could become significant and thus is one of the reasons I cannot convert it to non-recursive now.

Given that po/ is handled by gettext, is there any possibility of incorporating po/ into the non-recursive scheme? I assume the answer is "no".

The answer is no. GNU Bison code, which I take for inspiration for the non-recursive makefiles like this, still has a recursive makefile for the "po" directory.

[Mightyjo]

I'm referring to the stage1scan.c naming scheme. You've chosen not to change it back.

Your rationale for respecting LFLAGS is wrong. We are building flex, not executing flex. If you wanted deterministic builds, letting users change the build flags won't get you there.

You aren't using VPATH correctly in doc/ so you've had to copy the binary. You also didn't notice that users don't regenerate the docs so you're trying to solve a problem that doesn't (or shouldn't) exist.

You removed the ability to build in the src/, doc/, and examples/ directories. That was useful for unit testing the build system.

On Sat, May 11, 2024, 07:38 Kang-Che Sung (宋岡哲) @.***> wrote:

@Mightyjo https://github.com/Mightyjo

I understand the issue that you're having with the doc directory, but it comes back to the misunderstanding we were discussing regarding the name of the flex binary at various stages of the bootstrap and build process. Let me take a crack at fixing the libtoolized make files before we go down this path.

Excuse me. What misunderstanding?

(Just to attach the flex bootstrap diagram again for reference:) e9e9b93e-9e3f-11e6-8319-e4e33d38049b.png (view on web) https://github.com/westes/flex/assets/2487263/5faacd54-8718-4cf5-aed9-0fb411299cb1

— Reply to this email directly, view it on GitHub https://github.com/westes/flex/pull/652#issuecomment-2105688476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVJXII6HYZY2D6G7AGONQLZBX7MFAVCNFSM6AAAAABHRHOK46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGY4DQNBXGY . You are receiving this because you were mentioned.Message ID: @.***>

[Explorer09]

@Mightyjo

I'm referring to the stage1scan.c naming scheme. You've chosen not to change it back.

It's not my call for changing that naming, and I don't know what better name people would suggest then.

Your rationale for respecting LFLAGS is wrong. We are building flex, not executing flex. If you wanted deterministic builds, letting users change the build flags won't get you there.

It's deterministic with respect to user's build flags now. Perhaps you were confused about the definition of "deterministic". I didn't mean the scanner the user generated (stage1scan.c) would be always identical to the stock one (scan.c) provided in the distribution, but the generated scanner (stage1scan.c) can reproduce itself (stage2scan.c) with the just built "flex" generator.

A build is considered "deterministic" when, given the same source code, same compiler tools, and same build configurations (including CFLAGS and configure script options), then any person can produce bit-identical binaries regardless the machine they used. Build determinism mean eliminating non-deterministic factors such as system time, hostname, user names, and build paths (ideally, but some build paths can be tolerated to slip in binaries if they help debugging). Build determinism is a factor on whether users can trust the binaries especially for binaries from OS distributions. People don't want software backdoors or compiler viruses. (Compiler viruses are especially hard to uncover unless some kind of build determinism is forced.)

You aren't using VPATH correctly in doc/ so you've had to copy the binary.

Nope. VPATH is nothing to do with this. The problem was the different name of flex binaries: stage1flex and flex. Are you suggesting a solution to build the stage1 flex in a different directory, say, $(top_builddir)/stage1flex/flex$(EXEXT), so that the program name (within the directory) can be fixed like that? That way we can avoid copying the binary.

You also didn't notice that users don't regenerate the docs so you're trying to solve a problem that doesn't (or shouldn't) exist.

In CI servers the docs are frequency regenerated during build tests. Making the docs generated only during make dist won't cut it.

You removed the ability to build in the src/, doc/, and examples/ directories. That was useful for unit testing the build system.

There is nothing to build in the examples/ directory. For src/ and doc/, however, inter-directory dependencies would make just rebuilding a single directory of them a bad idea. If it was not these inter-directory dependencies, I would keep the makefiles as is and not suggest refactoring to non-recursive.

[Mightyjo]

@Mightyjo

I'm referring to the stage1scan.c naming scheme. You've chosen not to change it back.

It's not my call for changing that naming, and I don't know what better name people would suggest then.

You picked the current current naming. You could change it back. I suggest:

• Stage 1: sources and bins built from scan.l using previous version of Flex. I.e. stage1scan.c, stage1flex
• Dist: sources and bins built from stage1flex, I.e. scan.c, flex
• Stage 2: sources built by Flex in the build tree for validation. I.e. stage2scan.c

Your rationale for respecting LFLAGS is wrong. We are building flex, not executing flex. If you wanted deterministic builds, letting users change the build flags won't get you there.

It's deterministic with respect to user's build flags now.

Ignoring determinism for now. The users should not be determining the LFLAGS used to build Flex itself.

Perhaps you were confused about the definition of "deterministic". I didn't mean the scanner the user generated (stage1scan.c) would be always identical to the stock one (scan.c) provided in the distribution, but the generated scanner (stage1scan.c) can reproduce itself (stage2scan.c) with the just built "flex" generator.

That has been confusing. Thank you for defining your terms. "Determinism" carries a connotation that the steps to reach the end are important. You're more concerned that the end result is always the same. That's been difficult to reconcile with your goal of increasing parallelism.

[snip]

You aren't using VPATH correctly in doc/ so you've had to copy the binary.

Nope. VPATH is nothing to do with this. The problem was the different name of flex binaries: stage1flex and flex. Are you suggesting a solution to build the stage1 flex in a different directory, say, $(top_builddir)/stage1flex/flex$(EXEXT), so that the program name (within the directory) can be fixed like that? That way we can avoid copying the binary.

Yep. That's what I'm saying you should have done instead of copying the binary into doc/. Not doing so was "using VPATH incorrectly."

You also didn't notice that users don't regenerate the docs so you're trying to solve a problem that doesn't (or shouldn't) exist.

In CI servers the docs are frequency regenerated during build tests. Making the docs generated only during make dist won't cut it.

Wrong. This works fine, since make distcheck is the CI target.

You removed the ability to build in the src/, doc/, and examples/ directories. That was useful for unit testing the build system.

There is nothing to build in the examples/ directory.

Incorrect. The examples are in the examples/ directory. We direct users there to build and run the examples in the manual and when providing support.

For src/ and doc/, however, inter-directory dependencies would make just rebuilding a single directory of them a bad idea. If it was not these inter-directory dependencies, I would keep the makefiles as is and not suggest refactoring to non-recursive.

Explain why this is a bad idea.

[Explorer09]

@Mightyjo

I'm referring to the stage1scan.c naming scheme. You've chosen not to change it back.

It's not my call for changing that naming, and I don't know what better name people would suggest then.

You picked the current current naming. You could change it back. I suggest:

* Stage 1: sources and bins built from scan.l using previous version of Flex. I.e. stage1scan.c, stage1flex

* Dist: sources and bins built from stage1flex, I.e. scan.c, flex

* Stage 2: sources built by Flex in the  build tree for validation. I.e. stage2scan.c

I would not say the word "back" since the naming has been there when you made the PR about the bootstrap (#20).

"Determinism" carries a connotation that the steps to reach the end are important. You're more concerned that the end result is always the same. That's been difficult to reconcile with your goal of increasing parallelism.

Well, it's me who was addressing the problem on how to reach build parallelism and portability and the so called "determinism" at the same time. (And I almost got a way to do it.)

[snip]

You aren't using VPATH correctly in doc/ so you've had to copy the binary.

Nope. VPATH is nothing to do with this. The problem was the different name of flex binaries: stage1flex and flex. Are you suggesting a solution to build the stage1 flex in a different directory, say, $(top_builddir)/stage1flex/flex$(EXEXT), so that the program name (within the directory) can be fixed like that? That way we can avoid copying the binary.

Yep. That's what I'm saying you should have done instead of copying the binary into doc/. Not doing so was "using VPATH incorrectly."

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

Which directory would you suggest to put the stage1flex object files in? You can use your proposed naming scheme here, for convenience.

1. "$(top_builddir)/stage1flex" so that object files become "$(top_builddir)/stage1flex/{*.o,*.lo,*.obj}" and executable becomes "$(top_builddir)/stage1flex/flex{,.exe}"

2. "$(top_builddir)/src/stage1flex" so that object files become "$(top_builddir)/src/stage1flex/{*.o,*.lo,*.obj}" and executable becomes "$(top_builddir)/src/stage1flex/flex{,.exe}"

There is nothing to build in the examples/ directory.

Incorrect. The examples are in the examples/ directory. We direct users there to build and run the examples in the manual and when providing support.

Just to remind you that I didn't touch the makefile within the examples/manual subdirectory. It's still there, which is named "example/manual/Makefile.example" and same as before.

[Mightyjo]

I'm referring to the stage1scan.c naming scheme. You've chosen not to change it back.

It's not my call for changing that naming, and I don't know what better name people would suggest then.

You picked the current current naming. You could change it back. I suggest:

* Stage 1: sources and bins built from scan.l using previous version of Flex. I.e. stage1scan.c, stage1flex

* Dist: sources and bins built from stage1flex, I.e. scan.c, flex

* Stage 2: sources built by Flex in the  build tree for validation. I.e. stage2scan.c

I would not say the word "back" since the naming has been there when you made the PR about the bootstrap (#20).

"Determinism" carries a connotation that the steps to reach the end are important. You're more concerned that the end result is always the same. That's been difficult to reconcile with your goal of increasing parallelism.

Well, it's me who was addressing the problem on how to reach build parallelism and portability and the so called "determinism" at the same time. (And I almost got a way to do it.)

Meh. I could run make -j 8 just fine before.

You aren't using VPATH correctly in doc/ so you've had to copy the binary.

Nope. VPATH is nothing to do with this. The problem was the different name of flex binaries: stage1flex and flex. Are you suggesting a solution to build the stage1 flex in a different directory, say, $(top_builddir)/stage1flex/flex$(EXEXT), so that the program name (within the directory) can be fixed like that? That way we can avoid copying the binary.

Yep. That's what I'm saying you should have done instead of copying the binary into doc/. Not doing so was "using VPATH incorrectly."

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

Yes. This is what I intend. Also, I only intended for scan.c to go into the dist tarball. Seems that was malfunctioning. How do you have it working?

Which directory would you suggest to put the stage1flex object files in? You can use your proposed naming scheme here, for convenience.

1. "$(top_builddir)/stage1flex" so that object files become "$(top_builddir)/stage1flex/{*.o,*.lo,*.obj}" and executable becomes "$(top_builddir)/stage1flex/flex{,.exe}"
2. "$(top_builddir)/src/stage1flex" so that object files become "$(top_builddir)/src/stage1flex/{*.o,*.lo,*.obj}" and executable becomes "$(top_builddir)/src/stage1flex/flex{,.exe}"

I don't put these objects in subdirectories. I use the automake/libtool prefixing to keep them all together in srcdir and builddir. That's how I avoided recursive Make modules up to this point.

There is nothing to build in the examples/ directory.

Incorrect. The examples are in the examples/ directory. We direct users there to build and run the examples in the manual and when providing support.

Just to remind you that I didn't touch the makefile within the examples/manual subdirectory. It's still there, which is named "example/manual/Makefile.example" and same as before.

Good.

[Explorer09]

@Mightyjo

Well, it's me who was addressing the problem on how to reach build parallelism and portability and the so called "determinism" at the same time. (And I almost got a way to do it.)

Meh. I could run make -j 8 just fine before.

Well, I have explained why make -j 8 worked in the current, recursive scheme in a previous comment:

GNU make launches subdirectory makefiles one at the time, reducing parallelism to prevent inter-directory data contentions. And it invokes subdirectory makefiles in the order set by developers (in the SUBDIRS variable) and never out of order because the order dependencies are sometime not specified (developers know them but 'make' won't know them).

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

Yes. This is what I intend. Also, I only intended for scan.c to go into the dist tarball. Seems that was malfunctioning. How do you have it working?

I can override the scan.c rule pregenerated by Automake, but this is not done yet. Notice that Automake generated "scan.l to scan.c" rule would gotta respect user's LFLAGS anyway. So I cannot make it ignore LFLAGS during the build. (The scan.c provided in distribution can only be used when LFLAGS is empty or --disable-bootstrap is specified).

Which directory would you suggest to put the stage1flex object files in? You can use your proposed naming scheme here, for convenience.

I don't put these objects in subdirectories. I use the automake/libtool prefixing to keep them all together in srcdir and builddir. That's how I avoided recursive Make modules up to this point.

I have to make a correction in the last comment. Automake would kind of force me to put the stage1flex object (.o,.obj) files in the same directory as the source (.c) files. So I won't have that decision.

And it becomes tricky to me, that Automake didn't document the exact scheme how the object file name prefixes are derived, so I might not be permitted to use the same executable file names (despite different directories) in Automake. Oh no...

Just to remind you that I didn't touch the makefile within the examples/manual subdirectory. It's still there, which is named "example/manual/Makefile.example" and same as before.

Good.

I can add that, since the Makefile.am in that "examples/manual" directory go away, the example makefile can now occupy the main "Makefile" name without problem.

[Mightyjo]

@Mightyjo

Well, it's me who was addressing the problem on how to reach build parallelism and portability and the so called "determinism" at the same time. (And I almost got a way to do it.)

Meh. I could run make -j 8 just fine before.

Well, I have explained why make -j 8 worked in the current, recursive scheme in a previous comment:

GNU make launches subdirectory makefiles one at the time, reducing parallelism to prevent inter-directory data contentions. And it invokes subdirectory makefiles in the order set by developers (in the SUBDIRS variable) and never out of order because the order dependencies are sometime not specified (developers know them but 'make' won't know them).

This project was structured with all the executable and library files in the same directory. They're all handled by the same Makefile. Thus, Make is able to build the target objects in parallel. The tests are built after the main executables because it doesn't make any sense to build them sooner. Likewise, the documentation is built after the tests pass. You're arguing for parallelism where it isn't needed.

I understand the argument for non-recursive make. But go re-read Recursive Make Considered Harmful and you'll see this project is already following those suggestions. Miller wasn't talking about recursion related to documentation and tests. He was arguing against separating libraries and functional code modules into directories build by recursive Makefiles. Flex doesn't do that.

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

Yes. This is what I intend. Also, I only intended for scan.c to go into the dist tarball. Seems that was malfunctioning. How do you have it working?

I can override the scan.c rule pregenerated by Automake, but this is not done yet. Notice that Automake generated "scan.l to scan.c" rule would gotta respect user's LFLAGS anyway. So I cannot make it ignore LFLAGS during the build. (The scan.c provided in distribution can only be used when LFLAGS is empty or --disable-bootstrap is specified).

Go read the Automake and Libtool manuals all the way through again. Specifying any of the _*FLAGS per-target variables overrides the user-defined versions of those variables. E.g. defining stage1flex_CFLAGS causes CFLAGS to go unapplied unless we specifically include it in the per-target definition. I don't think specifying target_CFLAGS overrides LFLAGS, but I need to check.

@westes Reiterating my contention: It's incorrect to let users specify LFLAGS for scan.l when we're building Flex. At least when we're building the bootstrap scanner. What do you say?

Which directory would you suggest to put the stage1flex object files in? You can use your proposed naming scheme here, for convenience.

I don't put these objects in subdirectories. I use the automake/libtool prefixing to keep them all together in srcdir and builddir. That's how I avoided recursive Make modules up to this point.

I have to make a correction in the last comment. Automake would kind of force me to put the stage1flex object (.o,.obj) files in the same directory as the source (.c) files. So I won't have that decision.

LOL, you're just wrong. "So I won't have that decision." You're talking nonsense.

And it becomes tricky to me, that Automake didn't document the exact scheme how the object file name prefixes are derived, so I might not be permitted to use the same executable file names (despite different directories) in Automake. Oh no...

That's because the prefixes are a Libtool extension. I don't believe you've been reading the links I sent you. Read the Automake and Libtool manuals again.

Just to remind you that I didn't touch the makefile within the examples/manual subdirectory. It's still there, which is named "example/manual/Makefile.example" and same as before.

Good.

I can add that, since the Makefile.am in that "examples/manual" directory go away, the example makefile can now occupy the main "Makefile" name without problem.

[Explorer09]

@Mightyjo

Please, don't tell me to read the manual when I already did it. When I wrote this PR, I did a lot of build tests to make sure everything works within Automake, and sometimes I even rely on undocumented stuff in Automake.

You won't see where the build breaks until you test them out yourself and analyze the generated makefiles. Here is one of them, if you're interested: https://github.com/Explorer09/flex/actions/runs/9045080132

I understand the argument for non-recursive make. But go re-read Recursive Make Considered Harmful and you'll see this project is already following those suggestions. Miller wasn't talking about recursion related to documentation and tests. He was arguing against separating libraries and functional code modules into directories build by recursive Makefiles. Flex doesn't do that.

The doc/ and src/ directories are not completely separate in dependencies. The "help2man" part is the problem here.

"make -C doc/ flex.1" -> flex.1
                            |
                            |
                            V
"make -C src/ flex" -----> flex (program)

I mentioned in a previous comment (to @westes in particular), there is a data contention in flex build tree where two "make" instances can try to rebuild the same file at the same time, potentially breaking things. Whether it is about source files or documentation is irrelevant. As long as there is dependencies that resembles a diamond problem, the recursive makefiles would become a bad idea.

Yes. This is what I intend. Also, I only intended for scan.c to go into the dist tarball. Seems that was malfunctioning. How do you have it working?

I can override the scan.c rule pregenerated by Automake, but this is not done yet. Notice that Automake generated "scan.l to scan.c" rule would gotta respect user's LFLAGS anyway. So I cannot make it ignore LFLAGS during the build. (The scan.c provided in distribution can only be used when LFLAGS is empty or --disable-bootstrap is specified).

Go read the Automake and Libtool manuals all the way through again. Specifying any of the _*FLAGS per-target variables overrides the user-defined versions of those variables. E.g. defining stage1flex_CFLAGS causes CFLAGS to go unapplied unless we specifically include it in the per-target definition. I don't think specifying target_CFLAGS overrides LFLAGS, but I need to check.

No. "target_CLAGS" only override AM_CLAGS, not CFLAGS. I even dumped out the makefiles to verify the fact. (An example: https://github.com/Explorer09/flex/actions/runs/9045080132/job/24854430365)

I have to make a correction in the last comment. Automake would kind of force me to put the stage1flex object (.o,.obj) files in the same directory as the source (.c) files. So I won't have that decision.

LOL, you're just wrong. "So I won't have that decision." You're talking nonsense.

That's because the prefixes are a Libtool extension. I don't believe you've been reading the links I sent you. Read the Automake and Libtool manuals again.

Did you look at the makefiles Automake generated for you? I can know what works and what doesn't. When I name the program "stage1flex/flex", and with a program specific CFLAGS (stage1flex_flex_CFLAGS), Automake generates object file names like this: "src/stage1flex_flex-*.o"

Either in the src/ subdirectory of the top build directory, or in the top build directory itself (depending on "subdir-objects" automake option). I know this. The generated object files cannot relocate to an any directory of my choice, within Automake's scheme.

[Explorer09]

@Mightyjo

For src/ and doc/, however, inter-directory dependencies would make just rebuilding a single directory of them a bad idea. If it was not these inter-directory dependencies, I would keep the makefiles as is and not suggest refactoring to non-recursive.

Explain why this is a bad idea.

Imagine this scenario:

1. Users has a pregenerated "doc/flex.1" man page, potentially from a distribution tarball.
2. User edits src/main.c and other source code to add one command-line option with descriptions fo flex.
3. make -C doc flex.1
4. If the doc/Makefile.am doesn't come with this change (03b5af1572a611f3a2064119034d7201941d279e), the above "make" invocation won't update the flex program to generate the updated man page.

The 03b5af1572a611f3a2064119034d7201941d279e change is only a partial solution to the inter-directory dependency problem like this. If a makefile in the doc/ directory can update something in the src/ directory, then the makefiles would not be "independent" as you might think when it come to unit tests.

I hope this explains why. I didn't go non-recursive for the sake of it. And I know where non-recursive makefiles would not fit, as mentioned in a previous comment before.

[Mightyjo]

@Explorer09

Please, don't tell me to read the manual when I already did it. When I wrote this PR, I did a lot of build tests to make sure everything works within Automake, and sometimes I even rely on undocumented stuff in Automake.

Respect gets respect. Throughout our discussions I've gotten the strong impression that you either don't strongly dislike Automake and Libtool or just don't write in them often. This PR reinforces that belief. I'll acknowledge that I feel a strong ownership for the current build system even though lots of people have contributed to it. So that's part of my reaction. Beyond that, your explanations for why you want to make this change don't seem valid. Making the Makefile marginally smaller; following patterns that we can follow in either build system; supporting a build sequence that you think is more correct. I trust you and respect your coding ability. I don't think these recent build system changes are better than what we already had. Being dismissive isn't helping.

You won't see where the build breaks until you test them out yourself and analyze the generated makefiles. Here is one of them, if you're interested: https://github.com/Explorer09/flex/actions/runs/9045080132

What are you saying is wrong in this?

I understand the argument for non-recursive make. But go re-read Recursive Make Considered Harmful and you'll see this project is already following those suggestions. Miller wasn't talking about recursion related to documentation and tests. He was arguing against separating libraries and functional code modules into directories build by recursive Makefiles. Flex doesn't do that.

The doc/ and src/ directories are not completely separate in dependencies. The "help2man" part is the problem here.

Not the point I'm making. Those directories are meant to be built sequentially. They don't fall into the "recursive" definition in Miller

"make -C doc/ flex.1" -> flex.1
                            |
                            |
                            V
"make -C src/ flex" -----> flex (program)

Is this diagram what's happening now? flex.1 (via help2man) depends on the flex binary, not the other way around.

I mentioned in a previous comment (to @westes in particular), there is a data contention in flex build tree where two "make" instances can try to rebuild the same file at the same time, potentially breaking things.

When did that behavior start? Please find the commit so we can understand the regression because I last cleaned that up when I got ESR's commits ready for merge. I think you posted a log of it before but I couldn't expand it for some reason. Try again?

Whether it is about source files or documentation is irrelevant.

In this tree, the subdirectory is relevant. Source and documentation builds should be sequential, not parallel, so diamond dependencies aren't every a problem. The timing is known before build.

Yes. This is what I intend. Also, I only intended for scan.c to go into the dist tarball. Seems that was malfunctioning. How do you have it working?

I can override the scan.c rule pregenerated by Automake, but this is not done yet. Notice that Automake generated "scan.l to scan.c" rule would gotta respect user's LFLAGS anyway. So I cannot make it ignore LFLAGS during the build. (The scan.c provided in distribution can only be used when LFLAGS is empty or --disable-bootstrap is specified).

Go read the Automake and Libtool manuals all the way through again. Specifying any of the _*FLAGS per-target variables overrides the user-defined versions of those variables. E.g. defining stage1flex_CFLAGS causes CFLAGS to go unapplied unless we specifically include it in the per-target definition. I don't think specifying target_CFLAGS overrides LFLAGS, but I need to check.

No. "target_CLAGS" only override AM_CLAGS, not CFLAGS. I even dumped out the makefiles to verify the fact. (An example: https://github.com/Explorer09/flex/actions/runs/9045080132/job/24854430365)

Right, my mistake. So why did we ever need to worry about overriding LFLAGS in the first place? Unless it's in the bootstrap rules where I was overriding them specifically because we don't want the users changing our expectations of how the the new scan.c is made from scan.l?

I have to make a correction in the last comment. Automake would kind of force me to put the stage1flex object (.o,.obj) files in the same directory as the source (.c) files. So I won't have that decision.

LOL, you're just wrong. "So I won't have that decision." You're talking nonsense. That's because the prefixes are a Libtool extension. I don't believe you've been reading the links I sent you. Read the Automake and Libtool manuals again.

Did you look at the makefiles Automake generated for you? I can know what works and what doesn't. When I name the program "stage1flex/flex", and with a program specific CFLAGS (stage1flex_flex_CFLAGS), Automake generates object file names like this: "src/stage1flex_flex-*.o"

Yes, I read them frequently when I'm working on the build system. That's one reason I prefer them to be scoped to the relevant subdirectories. Automake and Libtool generate those names during compilation, then Libtool demangles them at link. I like that. You clearly don't. It's a bad reason to throw one system out for another either way.

Either in the src/ subdirectory of the top build directory, or in the top build directory itself (depending on "subdir-objects" automake option). I know this. The generated object files cannot relocate to an any directory of my choice, within Automake's scheme.

That's not the same place as the flex source files. Built sources and built objects being together in your builddir are fine.

[Mightyjo]

For src/ and doc/, however, inter-directory dependencies would make just rebuilding a single directory of them a bad idea. If it was not these inter-directory dependencies, I would keep the makefiles as is and not suggest refactoring to non-recursive.

Explain why this is a bad idea.

Imagine this scenario:

1. Users has a pregenerated "doc/flex.1" man page, potentially from a distribution tarball.
2. User edits src/main.c and other source code to add one command-line option with descriptions fo flex.
3. make -C doc flex.1
4. If the doc/Makefile.am doesn't come with this change (03b5af1), the above "make" invocation won't update the flex program to generate the updated man page.

The 03b5af1 change is only a partial solution to the inter-directory dependency problem like this. If a makefile in the doc/ directory can update something in the src/ directory, then the makefiles would not be "independent" as you might think when it come to unit tests.

I hope this explains why. I didn't go non-recursive for the sake of it. And I know where non-recursive makefiles would not fit, as mentioned in a previous comment before.

Changing the docs is a maintainer-level workflow. It's documented in README, INSTALL, and CONTRIBUTING. Assuming the user started from a dist tarball:

1. User installs the tools needed for maintenance
2. User runs make maintainer-clean
3. User write their patch
4. User runs ./autogen.sh && ./configure && make && make distcheck
5. User distributes and installs from their new tarball.

[Explorer09]

@Mightyjo

Respect gets respect. Throughout our discussions I've gotten the strong impression that you either don't strongly dislike Automake and Libtool or just don't write in them often. This PR reinforces that belief. I'll acknowledge that I feel a strong ownership for the current build system even though lots of people have contributed to it. So that's part of my reaction. [...] I don't think these recent build system changes are better than what we already had. Being dismissive isn't helping.

Can we have a discussion where we respect the work of each other? I just want to improve the flex build scheme of whatever it has right now, so I don't want to say anything about whether I like Automake or Libtool or not.

You won't see where the build breaks until you test them out yourself and analyze the generated makefiles. Here is one of them, if you're interested: https://github.com/Explorer09/flex/actions/runs/9045080132

What are you saying is wrong in this?

It's one of the build that I managed to make it work. And notice the eight build jobs there, testing the native build and cross build, with and without VPATH, and on different versions of Ubuntu distro. (Ubuntu 22.04 has an updated Autoconf, that uncovered a bug of ax_prog_cc_for_build, which I fixed in #649.)

The doc/ and src/ directories [...]

Not the point I'm making. Those directories are meant to be built sequentially. They don't fall into the "recursive" definition in Miller

Quote the definition from Miller, just for reference:

["Recursive make"] refers to the use of a hierarchy of directories containing source files for the modules which make up the project, where each of the sub-directories contains a Makefile which describes the rules and instructions for the make program. The complete project build is done by arranging for the top-level Makefile to change directory into each of the sub-directories and recursively invoke make.

So as long as the top-level makefile invokes the makefiles subdirectory makefiles with another instance of make it's the "recursive" definition Miller was talking about.

And yes. The subdirectory makefiles are invoked in sequence specified by the developers. This doesn't invalidate my point: make doesn't know why the subdirectories have to be in this order; it only knows the order itself. And it has to invoke the makefiles sequentially because there is no other way with the "recursive" scheme.

"make -C doc/ flex.1" -> flex.1
                            |
                            |
                            V
"make -C src/ flex" -----> flex (program)

Keep in mind the arrow in this diagram mean differently from the "bootstrap" diagram I given previously. Here, A --> B means "A depends on B" and that "A would trigger the rebuild of B".

Thus I hope you understand what could happen: Two make instances could trigger the build of flex (program) at the same time, touching the same file without mutexes guarding around.

And also regarding the scenario I mentioned:

1. Users has a pregenerated "doc/flex.1" man page, potentially from a distribution tarball.
2. User edits src/main.c and other source code to add one command-line option with descriptions fo flex.
3. "make -C doc flex.1"

Saying it's a maintainer level workflow won't cut it. Maybe it's just me, but I wish the documentation can be rebuilt every time from a git repo checkout. That's also what CI is testing, after all.

If the documentation is reproducible, it wouldn't hurt if users build it and build again and again, expecting the results be bit-for-bit identical to the original.

(And yes I know this requires help2man to be installed on the build system. It's easy to detect this in "configure", in my opinion.)

No. "target_CLAGS" only override AM_CLAGS, not CFLAGS.

Right, my mistake. So why did we ever need to worry about overriding LFLAGS in the first place?

The point is, scanners and parsers generated from ".l" and ".y" would always respect LFLAGS and YFLAGS during build. It's not maintainers who would rebuild the scanner and parser, but anyone who build from a checked-out git repository would gonna build them anyway, with the LFLAGS and YFLAGS.

This assumption comes from Automake. It wouldn't make sense, for Flex during bootstrap, to not honor LFLAGS.

And before you argue, I know what your vision is: Since you want scan.c to be included in the distribution, so that users don't need to rebuild the scanner C file (scan.c) and can skip the whole "stage1" process during normal build. My easy suggestion to this is making --disable-bootstrap option the default rather than --enable-bootstrap the default. When users wants the bootstrap, the assumption is LFLAGS should work, never ignored. I don't change that assumption.

[Explorer09]

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

@westes @Mightyjo

Not sure if you guys like discussing this further, but assuming the naming scheme like in this diagram, I have a few constraints:

• The entire source directory may be read-only during build. When user runs "configure" out of tree (aka. VPATH build in Automake terminology), nothing in the source directory should be deleted or updated. This mean any "scan.c" included in the dist tarball must not be modified during "make" phase.
• We shouldn't require user to have "flex" pre-installed if they configure and build from a dist tarball, even for --enable-bootstrap configuration. That means stage1scan.c in this diagram has to be included in dist tarball no matter what.
• The user can expect "stage1scan.c" and "scan.c" be different, if they build with custom LFLAGS. "scan.c" and "stage2scan.c" should be bit identical, as they are generated with the same LFLAGS. I could implement a byte comparison check between "stage1scan.c" and "scan.c", but that check should only perform when user LFLAGS is empty.

[Mightyjo]

Respect gets respect. Throughout our discussions I've gotten the strong impression that you either don't strongly dislike Automake and Libtool or just don't write in them often. This PR reinforces that belief. I'll acknowledge that I feel a strong ownership for the current build system even though lots of people have contributed to it. So that's part of my reaction. [...] I don't think these recent build system changes are better than what we already had. Being dismissive isn't helping.

Can we have a discussion where we respect the work of each other? I just want to improve the flex build scheme of whatever it has right now, so I don't want to say anything about whether I like Automake or Libtool or not.

I'd love to!

You won't see where the build breaks until you test them out yourself and analyze the generated makefiles. Here is one of them, if you're interested: https://github.com/Explorer09/flex/actions/runs/9045080132

What are you saying is wrong in this?

It's one of the build that I managed to make it work. And notice the eight build jobs there, testing the native build and cross build, with and without VPATH, and on different versions of Ubuntu distro. (Ubuntu 22.04 has an updated Autoconf, that uncovered a bug of ax_prog_cc_for_build, which I fixed in #649.)

Understood. I'll look again.

The doc/ and src/ directories [...]

Not the point I'm making. Those directories are meant to be built sequentially. They don't fall into the "recursive" definition in Miller

Quote the definition from Miller, just for reference:

["Recursive make"] refers to the use of a hierarchy of directories containing source files for the modules which make up the project, where each of the sub-directories contains a Makefile which describes the rules and instructions for the make program. The complete project build is done by arranging for the top-level Makefile to change directory into each of the sub-directories and recursively invoke make.

Thanks! Miller uses "module" in the sense of C program modules, groups of source files compiled into a single object file. He doesn't talk about anything else. Flex's build system isn't recursive according to his paper because all the modules with common source code reside in the same source directory and are built by one Makefile at the same level. Building a system of projects loosely-related projects in the same tree doesn't meet his definition.

So as long as the top-level makefile invokes the makefiles subdirectory makefiles with another instance of make it's the "recursive" definition Miller was talking about.

And yes. The subdirectory makefiles are invoked in sequence specified by the developers. This doesn't invalidate my point: make doesn't know why the subdirectories have to be in this order; it only knows the order itself. And it has to invoke the makefiles sequentially because there is no other way with the "recursive" scheme.

"make -C doc/ flex.1" -> flex.1
                            |
                            |
                            V
"make -C src/ flex" -----> flex (program)

Keep in mind the arrow in this diagram mean differently from the "bootstrap" diagram I given previously. Here, A --> B means "A depends on B" and that "A would trigger the rebuild of B".

I think this diagram is still wrong given your explanation. As described it should be:

 "make -C src/ flex" -----> flex (program)
                              |
                              |
                              V
"make -C doc/ flex.1" -> flex.1

(edited to fix the spacing here)

This will never result in two instances of make building flex (program) at once. It's possible that launching those commands in parallel shells can produce variable results for flex.1, but it's not a problem when running make distcheck.

Thus I hope you understand what could happen: Two make instances could trigger the build of flex (program) at the same time, touching the same file without mutexes guarding around.

If the diagram you provided is really what's happening there's a bug that needs attention right away. I will check.

And also regarding the scenario I mentioned:

1. Users has a pregenerated "doc/flex.1" man page, potentially from a distribution tarball.
2. User edits src/main.c and other source code to add one command-line option with descriptions fo flex.
3. "make -C doc flex.1"

Saying it's a maintainer level workflow won't cut it. Maybe it's just me, but I wish the documentation can be rebuilt every time from a git repo checkout. That's also what CI is testing, after all.

This is what package maintainers have expected since I started writing OSS. It's a pain to build RPMS, DPKG, etc. specs from excessively opinionated source trees. Trying to maintain a packaged Flex is what got me started working on its build system.

Docs are rebuilt every time from a git repo checkout, if you do the whole build the way we document it. That means:

./autogen.sh && ./configure && make && make distcheck

If the documentation is reproducible, it wouldn't hurt if users build it and build again and again, expecting the results be bit-for-bit identical to the original.

(And yes I know this requires help2man to be installed on the build system. It's easy to detect this in "configure", in my opinion.)

The trouble with having users build the docs every time is the requirement that they install a TeX system. We got a lot of complaints about that. Everyone wants the PDF target but so few want to install the pre-reqs. I'm not opposed to having the docs be regenerated on every build if we can find a tiny, tiny TeX system to depend upon that's available everywhere.

No. "target_CLAGS" only override AM_CLAGS, not CFLAGS.

Right, my mistake. So why did we ever need to worry about overriding LFLAGS in the first place?

The point is, scanners and parsers generated from ".l" and ".y" would always respect LFLAGS and YFLAGS during build. It's not maintainers who would rebuild the scanner and parser, but anyone who build from a checked-out git repository would gonna build them anyway, with the LFLAGS and YFLAGS.

This assumption comes from Automake. It wouldn't make sense, for Flex during bootstrap, to not honor LFLAGS.

See below.

And before you argue, I know what your vision is: Since you want scan.c to be included in the distribution, so that users don't need to rebuild the scanner C file (scan.c) and can skip the whole "stage1" process during normal build. My easy suggestion to this is making --disable-bootstrap option the default rather than --enable-bootstrap the default. When users wants the bootstrap, the assumption is LFLAGS should work, never ignored. I don't change that assumption.

We agree --disable-bootstrap being the better default. That's how it's supposed to work. I'll get that fixed. B

But that's not my argument. It's incorrect for a compiler build process to honor the flags meant for users of that compiler. Flex is compiler for scanners. Flex's build process should not make special effort to honor Flex user flags. Whether we include scan.c in the dist or not, we should be in total control of the flags used to build it.

[Mightyjo]

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

@westes @Mightyjo

Not sure if you guys like discussing this further, but assuming the naming scheme like in this diagram, I have a few constraints:

• The entire source directory may be read-only during build. When user runs "configure" out of tree (aka. VPATH build in Automake terminology), nothing in the source directory should be deleted or updated. This mean any "scan.c" included in the dist tarball must not be modified during "make" phase.

No argument here. The source directory is writable when using our documented build commands for the dist tarball, so it hasn't been a problem. make distcheck also does VPATH builds and that's been fine. If a user runs make maintainer-clean and wipes out the built sources, it's not our problem. They can untar again.

• We shouldn't require user to have "flex" pre-installed if they configure and build from a dist tarball, even for --enable-bootstrap configuration. That means stage1scan.c in this diagram has to be included in dist tarball no matter what.

Remind me what you mean by "bootstrap" again, please? I think we were close on this. I mean "regenerate scan.c from scan.l using a version of Flex built using the same version of scan.l (i.e. from stage1scan.c)."

Using my definition, we only need to include scan.c and the "other C source files" in the dist tarball. Users who want to generate a .c from a .l need some version of Lex installed. That's the nature of bootstrapping. Unless we include something like a stage0scan.c that we almost never change and use that to build the bootstrap flex binary. I don't think that would be better than having users install Flex from their package managers.

• The user can expect "stage1scan.c" and "scan.c" be different, if they build with custom LFLAGS.

(Reiterating my strong objection to allowing users to set LFLAGS while building Flex itself.) I've been thinking about this. It's possible for "stage1scan.c" and "scan.c" to be different anytime stage1scan.c is built with an older major version of Flex. When we change the skeletons it can make an impact on stage1scan.c. Building scan.c seems to eliminate the differences, as you'd expect. I remember seeing that once, but it's not caused a big problem over time.

"scan.c" and "stage2scan.c" should be bit identical, as they are generated with the same LFLAGS. I could implement a byte comparison check between "stage1scan.c" and "scan.c", but that check should only perform when user LFLAGS is empty.

I would love for the scan.c/stage2scan.c test to be part of make check. If writing it as a "log compiler" script is too annoying it could be part of make distcheck.

[Explorer09]

Thanks! Miller uses "module" in the sense of C program modules, groups of source files compiled into a single object file. He doesn't talk about anything else. Flex's build system isn't recursive according to his paper because all the modules with common source code reside in the same source directory and are built by one Makefile at the same level. Building a system of projects loosely-related projects in the same tree doesn't meet his definition.

The flex program has optional dependencies on malloc.c and realloc.c in the lib/ directory. Flex build system used to build "libcompat.la", a convenience library, that incorporates everything in the lib/ directory. (It was avoided after 713f5a0cb4ba5e4bfebae125edf4111fd8dae803.)

We were already hitting the problems because of recursive makefiles that you might not be aware of.

I know what scheme would be perfect for recursive makefiles, that would come with little or no dependency problem here: GCC source tree, with support libraries such as "libiberty", "libgcc", "libgfortran" individually configurable and buildable (i.e. they come with their own individial configure scripts). It would be still "recursive" as part of Miller's definition, but the dependencies would be "library-wide", not on individual source files.

A quick way to analyze whether the problems described in Miller can appear is to read the makefile in each subdirectory, and see if there is any file dependency with "../" in its path (except for library-wide dependencies, e.g. "../foo.a", that can be converted to "-lfoo" when necessary for linking).

I think this diagram is still wrong given your explanation. As described it should be:

 "make -C src/ flex" -----> flex (program)
                              |
                              |
                              V
"make -C doc/ flex.1" -> flex.1

(edited to fix the spacing here)

This will never result in two instances of make building flex (program) at once. It's possible that launching those commands in parallel shells can produce variable results for flex.1, but it's not a problem when running make distcheck.

You missed my point. The real point is: We cannot expect user to run "make -C src/ flex" before "make -C doc/ flex.1"! Unless we drop support of running "make" from the doc/ directory entirely.

In other words, drop support of this use case:

1. Users has a pregenerated "doc/flex.1" man page, potentially from a distribution tarball.
2. User edits src/main.c and other source code to add one command-line option with descriptions fo flex.
3. "make -C doc flex.1"

By removing doc/Makefile entirely.

The trouble with having users build the docs every time is the requirement that they install a TeX system. We got a lot of complaints about that. Everyone wants the PDF target but so few want to install the pre-reqs. I'm not opposed to having the docs be regenerated on every build if we can find a tiny, tiny TeX system to depend upon that's available everywhere.

Well, isn't "configure" able (or supposed) to detect whether user has "help2man" or "texi2any"?

That way I can tweak the doc/Makefile.am to do this:

if HAVE_HELP2MAN
flex.1: $(FLEX)
        $(HELP2MAN) \
        --name='$(PACKAGE_NAME)' \
        --section=1 \
        --source='The Flex Project' \
        --manual='Programming' \
        --output=$@ \
        $(FLEX)
else
# See if the man page needs updating. If it does, print a friendly error.
flex.1: $(srcdir)/configure.ac $(srcdir)/src/cpp-flex.skl $(srcdir)/src/options.c $(srcdir)/src/options.h
        @echo 'flex.1 man page is outdated; please install help2man and regenerate configure.'
        @false
endif

But that's not my argument. It's incorrect for a compiler build process to honor the flags meant for users of that compiler. Flex is compiler for scanners. Flex's build process should not make special effort to honor Flex user flags. Whether we include scan.c in the dist or not, we should be in total control of the flags used to build it.

It is the user who builds Flex, after all. When a compiler like gcc builds itself, it respects the user's CFLAGS, CXXFLAGS, and LDFLAGS as well. A package's developers have no say on how the user should build his own binaries, otherwise we could just ship the official binaries of the package and call it a day.

Remind me what you mean by "bootstrap" again, please? [...] I mean "regenerate scan.c from scan.l using a version of Flex built using the same version of scan.l (i.e. from stage1scan.c)."

Using my definition, we only need to include scan.c and the "other C source files" in the dist tarball. Users who want to generate a .c from a .l need some version of Lex installed. That's the nature of bootstrapping. Unless we include something like a stage0scan.c that we almost never change and use that to build the bootstrap flex binary. I don't think that would be better than having users install Flex from their package managers.

Any scan.c from the source directory cannot be modified during "make" process. If it comes from dist tarball then it has to remain there (unless user performs "make maintainer-clean" to delete it). That's the constraint.

We cannot say, ship scan.c within the dist tarball; allow user to overwrite it if he configures with "--enable-bootstrap". Nope, that won't work if user configures out of tree (i.e. VPATH build). With the aforementioned constraint, the best we could do is to not ship scan.c, but stage1scan.c only.

[Explorer09]

@Mightyjo

    scan.l -in->    $(FLEX) on
       |         builder's system
       |           |
       |           V out
       |         stage1scan.c
       |            |
       |            V in            other C
       |      $(CC_FOR_BUILD) <-in- source
       |           |                files
       |           V out            |
       +----in-> stage1flex         |
       |           |                |
       |           V out            V in
       |          scan.c -----in-> $(CC)
       |                            |
       |                            V out
       +----------------------in-> flex
                                    |
                                    V out
                              stage2scan.c

Here is a summary of the three scanner C files:

• stage1scan.c: May be either (a) generated by Flex on the builder's system, which may be an outdated version, or (b) provided by dist tarball if user configures and builds from a dist tarball. For (a), the user's LFLAGS is honored during build, but ultimately won't affect the final Flex binary. For (b) the pregenerated files uses no LFLAGS.
• scan.c: Generated by just-built stage1flex, so we can guarantee it's always the exact version of the release. It should honor the user's LFLAGS during build. If the user's LFLAGS is empty, the generated C code should be bit-for-bit identical to the pregenerated scanner in the dist tarball (stage1scan.c (b)).
• stage2scan.c: Generated by the final, just-built flex binary, with the same LFLAGS that is used for generating scan.c. This file should be bit-for-bit identical to scan.c without exceptions. If flex is configured to be cross-compiled, this stage2scan.c step would be skipped.

[Mightyjo]

Thanks! Miller uses "module" in the sense of C program modules, groups of source files compiled into a single object file. He doesn't talk about anything else. Flex's build system isn't recursive according to his paper because all the modules with common source code reside in the same source directory and are built by one Makefile at the same level. Building a system of projects loosely-related projects in the same tree doesn't meet his definition.

The flex program has optional dependencies on malloc.c and realloc.c in the lib/ directory. Flex build system used to build "libcompat.la", a convenience library, that incorporates everything in the lib/ directory. (It was avoided after 713f5a0.)

The Makefile in lib/ only handled distribution. There was never a danger of compilation races. With that removed we're only getting distribution on lib/ through the CROSS version of libfl_SOURCES. That's awfully fragile.

We were already hitting the problems because of recursive makefiles

Explain where. I keep asking and you haven't said.

that you might not be aware of.

This is rude.

[snip]

This was irrelevant

I think this diagram is still wrong given your explanation. As described it should be:

 "make -C src/ flex" -----> flex (program)
                              |
                              |
                              V
"make -C doc/ flex.1" -> flex.1

(edited to fix the spacing here) This will never result in two instances of make building flex (program) at once. It's possible that launching those commands in parallel shells can produce variable results for flex.1, but it's not a problem when running make distcheck.

You missed my point. The real point is: We cannot expect user to run "make -C src/ flex" before "make -C doc/ flex.1"! Unless we drop support of running "make" from the doc/ directory entirely.

You've changed the topic. We prescribe the command sequence users must run to consistently build Flex. It's in our INSTALL.md. Users can do anything with their source tree and we don't care. But we only promise we'll build consistently if they follow our instructions.

The trouble with having users build the docs every time is the requirement that they install a TeX system. We got a lot of complaints about that. Everyone wants the PDF target but so few want to install the pre-reqs. I'm not opposed to having the docs be regenerated on every build if we can find a tiny, tiny TeX system to depend upon that's available everywhere.

Well, isn't "configure" able (or supposed) to detect whether user has "help2man" or "texi2any"?

We already detect this in ./configure and issue warnings.

But that's not my argument. It's incorrect for a compiler build process to honor the flags meant for users of that compiler. Flex is compiler for scanners. Flex's build process should not make special effort to honor Flex user flags. Whether we include scan.c in the dist or not, we should be in total control of the flags used to build it.

It is the user who builds Flex, after all. When a compiler like gcc builds itself, it respects the user's CFLAGS, CXXFLAGS, and LDFLAGS as well. A package's developers have no say on how the user should build his own binaries, otherwise we could just ship the official binaries of the package and call it a day.

We're upstream maintainers. We are mainly supporting downstream package maintainers who do ship official binaries and call it a day. Doesn't seem like that with all the user support requests, but they're asking how to use Flex more than how to build it.

Which of these defaults do you want users to override for their personal versions of Flex? %option caseless nodefault noreject stack noyy_top_state %option nostdinit (default flags) -8 --emit=C99 -B (defult options) nomain

Remind me what you mean by "bootstrap" again, please? [...] I mean "regenerate scan.c from scan.l using a version of Flex built using the same version of scan.l (i.e. from stage1scan.c)." Using my definition, we only need to include scan.c and the "other C source files" in the dist tarball. Users who want to generate a .c from a .l need some version of Lex installed. That's the nature of bootstrapping. Unless we include something like a stage0scan.c that we almost never change and use that to build the bootstrap flex binary. I don't think that would be better than having users install Flex from their package managers.

Any scan.c from the source directory cannot be modified during "make" process. If it comes from dist tarball then it has to remain there (unless user performs "make maintainer-clean" to delete it). That's the constraint.

You haven't explained what you mean by "bootstrap". Please answer my question.

We cannot say, ship scan.c within the dist tarball; allow user to overwrite it if he configures with "--enable-bootstrap".

Like I said above. We prescribe the build commands that produce working binaries. Users can format their hard drives for all I care. It's not our problem to solve.

Nope, that won't work if user configures out of tree (i.e. VPATH build).

VPATH doesn't stop us from writing to srcdir. That's a distcheck constraint. It's not good practice, but we already mitigate it by delivering tarballs that won't require writing outside of the build directory. Again, we can't control users doing the Wrong Thing.

With the aforementioned constraint, the best we could do is to not ship scan.c, but stage1scan.c only.

Repeating myself for emphasis. You are trying to solve users doing the Wrong Thing. This isn't our problem. We document the supported build process. We provide CI that follows the supported build process. Done.

@westes I recommend we don't go down this path.

[Explorer09]

@Mightyjo

The Makefile in lib/ only handled distribution. There was never a danger of compilation races. With that removed we're only getting distribution on lib/ through the CROSS version of libfl_SOURCES. That's awfully fragile.

I have an impression that you didn't take a look at the current source tree of Flex, and you made wrong assumptions.

The lib/ directory of Flex source tree is NOT libfl. The directory contain two files that are compatibility functions that work like Gnulib. And I am now annoyed when you even get basic facts wrong.

[Explorer09]

https://github.com/Explorer09/flex/commit/4141ecf581ca915aa2c9d1a3d33edfc6f4cc27c8

https://github.com/Explorer09/flex/actions/runs/9052956528 https://github.com/Explorer09/flex/actions/runs/9053034878

[westes]

I would love for the scan.c/stage2scan.c test to be part of make check.

Would you please open that as a separate issue? (Who does the pr is independent.)

[westes]

Unless we include something like a stage0scan.c that we almost never change

I think this will lead to no end of heartache.

[westes]

--disable-bootstrap being the better default.

On the one hand, I'd love this to be the case as it makes it easier for most folks to build and use flex. On the other, those of us who routinely build from a clean git tree need it to not be the case during our normal operations.

Ideally, the source distributions would be packaged with this as the default. I have not given any thought to doing this, however.

[westes]

With respect to the man page:

As I understand it, the problem is that if cross-compiling, the built "flex" binary is built for a different system than the one being run on. Hence the mess with having to figure out what the name of the binary to run is. Also help2man doesn't have a way to say "run binary x but label the page y".

[Mightyjo]

--disable-bootstrap being the better default.

On the one hand, I'd love this to be the case as it makes it easier for most folks to build and use flex. On the other, those of us who routinely build from a clean git tree need it to not be the case during our normal operations.

Ideally, the source distributions would be packaged with this as the default. I have not given any thought to doing this, however.

I'm taking this for action. It's what I intended to happen all along.

[Mightyjo]

Unless we include something like a stage0scan.c that we almost never change

I think this will lead to no end of heartache.

Agreed. Just meant to list this option, not endorse it.

[Mightyjo]

I would love for the scan.c/stage2scan.c test to be part of make check.

Would you please open that as a separate issue? (Who does the pr is independent.)

Sure

[Mightyjo]

With respect to the man page:

As I understand it, the problem is that if cross-compiling, the built "flex" binary is built for a different system than the one being run on. Hence the mess with having to figure out what the name of the binary to run is. Also help2man doesn't have a way to say "run binary x but label the page y".

Looking into this. Seems the man page is built during make all so it should be possible to make the cross-compiled version pick up the right binary.

[westes]

Seems the man page is built during make all so it should be possible to make the cross-compiled version pick up the right binary.

Help2man needs the binary to have the name that will appear in the man page. That was the catch requiring the install command. (Or building in a dedicated directory is another way to solve that problem.)

[Explorer09]

I've added the last commit which shows a proof of concept of how a bootstrap can be done portably. And for the man pages, I made it so that both the stage1 program and the final binary to have the same file name to generate man page from. This is my ideal vision on how it can be done: 33d6725d6c39cab67354b653eff4b96bd3a34670

[Mightyjo]

Seems the man page is built during make all so it should be possible to make the cross-compiled version pick up the right binary.

Help2man needs the binary to have the name that will appear in the man page. That was the catch requiring the install command. (Or building in a dedicated directory is another way to solve that problem.)

I copy.

[Explorer09]

--disable-bootstrap being the better default.

On the one hand, I'd love this to be the case as it makes it easier for most folks to build and use flex. On the other, those of us who routinely build from a clean git tree need it to not be the case during our normal operations. Ideally, the source distributions would be packaged with this as the default. I have not given any thought to doing this, however.

I'm taking this for action. It's what I intended to happen all along.

@westes

How about this: check if "${top_srcdir}/src/scan.c" exists and is generated by the correct version of Flex, by verifying the predefined FLEX_*_VERSION C macros, all in configure time? If the file exists and the version check is correct, set "--disable-bootstrap" the default, otherwise, "--enable-bootstrap".

[westes]

That sounds correct so far as it goes.

A good test case is if you get the correct result based on the current master branch. That new release is coming but you guys keep posting interesting questions/issues/comments on these pr's. So the current state of those macros in scan.c probably pushes the boundary of what your test is supposed to uncover and in exactly the right way.