compiler: Option to produce a depfile from target dependencies

[CyberTailor]

Title

Depfile

Abstract

Add a new command-line option (e.g. --depfile) for the nim compile command. It accepts an argument (output path for a depfile, generated for a given target).

Motivation

This change would improve Nim integrations with language-independent build systems. Particularly, get better edit-compile cycle times.

Currently a build system needs to rebuild a binary if any of source files has been changed, even if they aren't used by the binary. The full list of files that a given source file depends on can only be discovered by the compiler.

It doesn't need to be a standalone command because if the file has never been compiled, it must be built anyway, generating dependencies as a side effect.

Description

Downside: growing number of command-line flags :)

Similar proposal: #412. However custom JSON format will not be understood by build systems.

Depfile file format

From GCC manual (-M option):

The preprocessor outputs one 'make' rule containing the object file name for that source file, a colon, and the names of all the included files, including those coming from '-include' or '-imacros' command-line options.

Or, in the BNF notation:
https://cmake.org/cmake/help/latest/command/add_custom_command.html

It is recognized by all common build systems.

Code Examples

Before

(no command to produce a depfile)

After

$ nim c --depfile main.d main
$ head -4 main.d # system imports not shown
main: \
    /home/project/main.nim \
    /home/project/src/file1.nim \
    /home/project/src/file2.nim \

Use the depfile in Makefile

main: main.nim
    nim c $(NIMFLAGS) --depfile $@.d $<

-include main.d

Use the depfile in Ninja

rule nimc:
  deps = gcc
  depfile = $out.d
  command = nim c $nimflags --depfile $out.d $in

build main: nimc main.nim

Use the depfile in CMake

...

add_custom_command(
    OUTPUT main
    COMMAND nim c ${NIMFLAGS} --depfile main.d main.nim
    DEPFILE main.d
    VERBATIM
)

Use the depfile in Meson

...

nim = find_program('nim')
custom_target(
  input : 'main.nim',
  output : 'main',
  depfile : '@BASENAME@.d',
  command : [nim, 'c', '--depfile', '@DEPFILE@', '@INPUT@'],
  build_by_default : true
)

Backwards Compatibility

This RFC introduces no backward-incompatible changes.

[Araq]

So write (and contribute!) a tool that translates the json file into whatever format that you need.

[CyberTailor]

So write (and contribute!) a tool that translates the json file into whatever format that you need.

I haven't found a command that generates file's dependencies. Does it exist?

[Araq]

The json file is always produced in nimcache.

[CyberTailor]

Hmm. nim c --genScript:on main generates such list of files (main.depend). So I guess a tool can be used to convert it to a depfile and we call it a day.

However if its format was standard depfile and it was allowed to set a custom output path for that file, things would be simpler.

[eli-schwartz]

Nonstandard formats will not be understood by ninja / cmake / meson unless you "write (and contribute!) a tool" that both runs the compiler and converts the nonstandard format.

Makefile rules are inherently embedded shell scripts, so you can write an inline Makefile wrapper script that first runs the compiler, then runs the tool, which isn't as bad. But this doesn't work generically.

If the goal is for the compiler to support "all common build systems" then there is no alternative and no half measure -- you need the compiler itself to support the standard format.

If the goal is for "all common build systems" to specifically support the compiler, then each build system would need to have a feature request opened + accepted + implemented to have the build system support the nonstandard format. This might prove challenging.

and it was allowed to set a custom output path for that file

While it is more flexible to support a custom output path for the depfile, it isn't really required. Neither ninja nor make nor cmake nor meson require that the depfile path appear as part of the compiler rule -- they merely require that the rule ultimately creates the depfile, and that the build system knows which path to look for the depfile at. For example,

• GCC: gcc -MD path/to/foo.c -c -o path/to/foo.o defaults to creating a depfile at path/to/foo.d.
  • can be overridden by -MF alternative/path/to/foo.o.d
• Qt moc: --output-dep-file which for output files like path/to/moc_foo.cpp defaults to create path/to/moc_foo.cpp.d.
  • can be overridden by --dep-file-path alternative/path/to/moc_foo.d

So:

• based on precedence, you do not need to have an option to specify the output file path
• strictly speaking, you do not need an option at all, you could always create a depfile unconditionally whether the user wants it or not. As long as the build system can predict where to find it. The build system won't know where to find it if it's not somewhere in the build tree, though, so if it's somewhere in $XDG_CACHE_HOME then --nimcache might be needed. It might be wanted regardless...

[CyberTailor]

Implemented in PR https://github.com/nim-lang/Nim/pull/19960

[arnetheduck]

https://github.com/status-im/nimbus-eth2/blob/unstable/tools/generate_makefile.nim - here's an example a tool that generates makefile rules from the json

uninteresting, per comment below

[zah]

There seems to be a misunderstanding here. The depfile needs to show a list of files that Nim read during the compilation process, not the list of compiled C sources. As a specific example, an included nim file should be part of the depfile, but it won't be mentioned at all in the json file. Slurped files, config files, etc are other build process inputs that belong to the depfile.

[zah]

I've just learned that the --run option produces a proper list of depfiles stored in the nimcache json, so it looks like it just needs to be wired to a separate new option.

[ringabout]

I've just learned that the --run option produces a proper list of depfiles stored in the nimcache json, so it looks like it just needs to be wired to a separate new option.

@zah Hi, -d:nimBetterRun exists, which produces proper depfiles and doesn't actually run the executable.