[csolution] `compile_commands.json` not referenced in generated `*.cbuild.yml`

[acabarbaye]

We ran cbuild with -t=database to generate the compile_commands.json from a csolution project. This file is not referenced in the generated cbuild.yml file as we would expect. See the files below hello.Debug+B-U585I-IOT02A.cbuild.yml.txt hello.cbuild-idx.yml.txt

As can be seen in the file, the output information mentions an hex file and an axf file which actually have not been generated:

  output-dirs:
    intdir: tmp/hello/B-U585I-IOT02A/Debug
    outdir: out/hello/B-U585I-IOT02A/Debug
    rtedir: RTE
  output:
    - type: elf
      file: hello.axf
    - type: hex
      file: hello.hex

[jkrech]

Tools reading the cbuild and cbuild-idx files should not make a blind assumption about output files and directories to exist. Just assume a build fails and an axf, lib, bin or hex file is not created. We will not go back and update the cbuild-idx file depending on the success of a build. Listing the compile_commands.json file as an output file should be feasible. @brondani what do you think?

[brondani]

It seems we should clarify and align expectations.

cbuild-idx.yml and cbuild.yml are the result of the project manager, not the result of the build manager. They are intended to be the input for the build manager (and in practice they will be the input once cprj will be fully replaced).

output-dirs and output can be read as the output locations and filenames that will be generated IF you build it.

Calling cbuild with target database is clearly not an artifact building task, it generates only the compile_commands.json. It seems this is still an undocumented feature, I guess we should extend the documentation to avoid such misunderstanding.

Generating compile commands is a pure build manager feature, the project manager currently doesn't know about it, so I wouldn't simply add it to project manager results, unless we decide to make the project manager aware of it.

[jkrech]

Thanks for clarifying.

[acabarbaye]

@brondani @jkrech Sorry I had completely missed your answers here. From our previous discussions, I understood that to retrieve artefacts produced by a job (whether it is artefacts from builds or for intellisense), I should read the cbuild-idx file to get the cbuild.yml file and from there find out what has been produced. From your answers here, I imagine I had misunderstood the purpose of those two files as they seem to be primarily for internal use of the build process itself and not for describing the result of a job. Is there a way from the output directory to determine what constitutes an output artefact rather than an intermediary build artefact?

[ReinhardKeil]

@acabarbaye these files are not just for internal use. They contain details about the build process. IMHO these are the files that the IDE should use to understand the location of build artifacts and individual sources files of a component.

[acabarbaye]

in that case, should not the compile_commands.json files generated for a project be mentioned there as they are the artefacts?

[ReinhardKeil]

@acabarbaye I agree, we need to make this more useful for IDE usage. Not sure if compile_commands.json should be a direct artifact of the CMSIS-Toolbox.

[acabarbaye]

they are the results of -t=database and are produced by CMSIS-Toolbox because they are toolchain specific and require project understanding

[brondani]

@acabarbaye I believe I understand your point, I would suggest to add it to cbuild.yml under the output node similarly as the build artifacts:

  output:
    - type: database
      file: compile_commands.json

But for consistency its presence/absence and the file value should be reflected in the CMakeLists.txt generation. I would suggest to handle this entry in the cbuild2cmake but not in the old cbuildgen since it would require cprj extensions.

[ReinhardKeil]

When we add it this way, the compile_commands.json file would be created also for stand-alone build systems (i.e. CI, MLOps).

I would recommend to build the IDE specific information as suggested here: https://github.com/Open-CMSIS-Pack/devtools/issues/1265#issuecomment-1887396374 - effectively with a command-line option.

[jkrech]

@ReinhardKeil - csolution is generating the cbuild.yml files. All specified output artifacts will only exist if the corresponding context is build. Therefore mentioning a hex or axf file in cbuild.yml is no guarantee that this file really exists. I think the same is true for compile_commands.json files.

[ReinhardKeil]

I don't believe this is required to reflect the compile_commands.json in *.cbuild.yml. The compile_commands.json is generated by CMake and therefore it is not that easy to add.

With cbuild setup the compile_commands.json file is generated in the out directory of the related context. This is now covered in documentation

[ReinhardKeil]

closing this as it appears to be a duplicate of #1426

[jkrech]

The action remaining from this issue is to add the references to the compile_commands.json to the *.cbuild-idx.yml file unless this is rejected.