Writing source dependencies to a file

[madig]

So, build tools like ninja have a way to instruct compilers to save a file with dependencies of whatever is being compiled (see https://ninja-build.org/manual.html#ref_headers), because it's easier to have the compiler output the files it touched than having to specify them up front. The logic is that the first compile needs to build everything anyway, but subsequent builds may not need to.

It would be nice if fontc had something similar :) that would make writing build system scripts more pleasant.

[rsheeter]

For context, fontc supports incremental compilation when IR is written to disk, including finer grained increments than whole files (crucial for .glyphs). For example, if you compile a .glyphs file, change one glyph, and compile the updated .glyphs file the compile should (modulo bugs) only rebuild that glyph and things that depend on that glyph such as the final glyf table, any glyph that uses that glyph as a component, etc.

We don't use separate executables and we don't explicitly write down the graph. The user isn't meant to have to care.

make writing build system scripts more pleasant.

Is "supports incremental compilation" all you need or is there some specific need to write down the dependency graph?

[madig]

My concern is build systems, which maintain their own dependency graph. Consider a rule like

font.ttf: my.designspace
    fontc $input -o $output

How would I get the build system to run fontc again when a constituent UFO source changes? fontc doesn't get a say in the matter at this level. I have to list all the underlying source files for the build system, somehow :/ This problem doesn't exist for self-contained .glyphs files, because updating it will tell the build system all it needs, but as soon as external feature files come into play, we're back again.

[rsheeter]

I see your point but I tend to think it would be weird for the compiler to take on this job.

What is it you are trying to build exactly, what scope does the "build system" take on?

Running with guesses at the answer to ^, perhaps you could have an orchestrator whose job it is to write ninja file? - we do that for nanoemoji and found it quite effective. Something that speaks designspace, e.g. norad, could presumably help to enumerate the files of a designspace.

It will also play out slightly oddly that both ninja and fontc want to handle concurrency. I suppose you'd want to decide who you want to do that and set limits on the other.

[madig]

Yes, I can discover file dependencies myself and write the build files. My thought was doing so is costly, and it would be nice if it's done only once, but alas.

[anthrotype]

let's keep this open. I think "playing nice" with existing build tools should be among the goals of fontc.

[anthrotype]

I think Rod's argument would be, since fontc already takes care of incremental builds and concurrency, one should not need to use a separate build tool like make or ninja to control fontc execution, but may simply run it directly, or if you have to use such build tool you make the target calling fontc a forced one or .PHONY such that it runs all the time, and then fontc takes care of rebuilding only if needed. Though it would be nice if fontc could optionally emit a depfile similarly to how gcc/clang do for C/C++ headers, especially when sources are scattered in different files like with designspaces or included feature files. Who else better than the font compiler itself knows what these relationships are for a given build target.

[madig]

The problem with always-run targets would be that all subsequent tasks would be run, too, defeating the purpose of a build system, unless you worked around that, somehow.

[rsheeter]

I wouldn't make the target always-run, I would make a tool to list sources and use that in the make rule, similar to how Noto Emoji builds. Ex:

FILES_IN_MY_DESIGNSPACE = $(ls_designspace my.designspace)
...
font.ttf: $(FILES_IN_MY_DESIGNSPACE)
    fontc --source my.designspace

https://github.com/googlefonts/noto-emoji/blob/main/Makefile does this sort of thing a fair bit.

[rsheeter]

Who else better than the font compiler itself knows what these relationships are for a given build target.

For a .glyphs file the answer from the pov of any tool that reasons about whole files is simple. For designspace ... is it common it's not just "all files in some directory"?

[madig]

Yes, just listing the files works. I'm interested in going to ninja-levels of efficiency by collecting implicit dependencies as a side-effect of just running the command :) It would simply be nice to avoid I/O where possible, especially on Windows.

[rsheeter]

Ninja is designed to work with a higher level tool that generates the ninja file but that doesn't seem like it's a job for fontc, rather I would expect ninja to run fontc. I imagine we'd want a tool to generate a ninja file, including materializing the list of files required to build the font. If that tool emits a file with directory mtimes then a nop build just reads those and confirms nothing changed. If a directory changed you rewrite the ninja file and update it if it changed before kicking off ninja.

Have you tried something along these lines and found it to be too much IO for Windows?

[madig]

I haven't done that, and my idea is to avoid doing that and use what ninja gives us.

Ninja would run fontc rather than the other way around, and remember the deps the compiler gives it for next time, so that the build recipe is just the input DS and output TTF and the command to get from one to the other. The ninja manual says

Consider: if the file has never been compiled, it must be built anyway, generating the header dependencies as a side effect. If any file is later modified (even in a way that changes which headers it depends on) the modification will cause a rebuild as well, keeping the dependencies up to date.

My understanding is that you run a generator once and then ninja figures out what it has to run on each invocation, similarly how you run cmake/meson setup once and then call cmake --build/meson compile after each source change. Therein lies my desire to signal to ninja which files it has to check, rather than enumerating them on each invocation. That would surely work, but it would waste IO.

[rsheeter]

I haven't done that, and my idea is to avoid doing that and use what ninja gives us.

Ninja would run fontc rather than the other way around

That is what I am suggesting also: you would have a tool to generate a ninja file that invokes fontc.

remember the deps the compiler gives it for next time

I believe this will trip on new files. If your ninja rule says something like a.glif and b.glif are dependencies then when you add c.glif and rebuild it may think nothing has changed. By having your ninja generator tool detect that it needs to rebuild the ninja file this case Just Works.

[rsheeter]

In re-reading I think a ninja-generator is a valid thing to want but its not part of fontc, rather it's a separate higher level thing that usees fontc.