`[csolution/cbuild]` all contexts present in cbuild-idx.yml even if a particular context was selected

[acabarbaye]

Describe the bug This is closely related to https://github.com/Open-CMSIS-Pack/devtools/issues/1210

cbuild/csolution were run for a particular context (here FRDM-K32L3A6.Debug+FRDM-K32L3A6):

the following is the content of the cbuild-idx.yml and all contexts are referenced; not just the one selected.

build-idx:
  generated-by: csolution version 2.2.1
  csolution: ../workspace/workspace/Blinky_FRDM-K32L3A6/Blinky.csolution.yml
  cprojects:
    - cproject: ../workspace/workspace/Blinky_FRDM-K32L3A6/FRDM-K32L3A6.cproject.yml
  cbuilds:
    - cbuild: FRDM-K32L3A6.Debug+FRDM-K32L3A6.cbuild.yml
      project: FRDM-K32L3A6
      configuration: .Debug+FRDM-K32L3A6
    - cbuild: FRDM-K32L3A6.Release+FRDM-K32L3A6.cbuild.yml
      project: FRDM-K32L3A6
      configuration: .Release+FRDM-K32L3A6

from previous unrelated builds, other context's artefacts are also present in the output folder and so, it is impossible to distinguish from the listing in the index file what artefacts are the output of the job we just ran and which ones are the result of previous jobs

Expected behavior Only list the data related to the context selected and job performed

Environment (please complete the following information):

cbuild 2.2

[jkrech]

For the sake of IDE integrations the concept of a "context set" was introduced. When you invoke cbuild with -S / --context-set as well as selecting the contexts to be built using c / --context in addition to the *.cbuild-idx.yml the file *.cbuild-set.yml is generated. It can be (re-)used in subsequent tool invocations for the same selection of contexts. As the *.cbuild-idx.yml file is always listing all contexts of a *.csolution.yml the *.cbuild-set.yml lists the contexts being updated during the last cbuild ... -S / csolution ... -S runs.

Does that solve the above problem or is it required to inspect timestamp of files?

[acabarbaye]

Just to clarify. .cbuild-idx.yml contains a list of paths to <context>.cbuild.yml for all the contexts defined in the csolution projects no matter what --context option was used and so, if this option is used, only corresponding <context>.cbuild.yml is created and therefore, the index file may reference files which do not exist. Moreover, if I run cbuild multiple times with different --context options, I can only know which context was build by looking at the timestamp of the <context>.cbuild.yml files and consider the ones which were created or modifield after I ran the cbuild command. `.cbuild-set.yml` has indeed only just been introduced and so, is not an option for older versions on cbuild 2.x which I need to support.

[brondani]

This is not a bug, it is intended behaviour: https://github.com/Open-CMSIS-Pack/devtools/issues/1125 (introduced in CMSIS-Toolbox 2.2.0

[acabarbaye]

Thank you @brondani for pointing me to this decision record. I had missed that completely. Could you confirm that my understanding of how context specific build artefacts should be retrieved is correct https://github.com/Open-CMSIS-Pack/devtools/issues/1426#issuecomment-2051395452 ?

Furthermore, I believe we should change the wording in the documentation https://github.com/Open-CMSIS-Pack/cmsis-toolbox/blob/main/docs/YML-CBuild-Format.md#file-structure-of-cbuild-idxyml to reflect that behaviour

cbuilds: List of *.cbuild.yml output files that are generated for this application.

should probably become something along the lines of

cbuilds: List of *.cbuild.yml files corresponding to the different contexts defined in the csolution project (Note: file may not be generated if the corresponding context is not selected)

[brondani]

Could you confirm that my understanding of how context specific build artefacts should be retrieved is correct #1426 (comment) ?

In the same way build artifacts are referenced in cbuild.yml but don't exist until a successful build is performed, it cannot be assumed a referenced cbuild.yml file exist until a successful convert call is performed. In addition it may exist but be incomplete since it is also used for error status reporting https://github.com/Open-CMSIS-Pack/devtools/issues/1385 and note that cbuild-idx.yml is also used for compatible layers reporting https://github.com/Open-CMSIS-Pack/devtools/issues/1379. I would say the information you can get from these files depends on the workflow you are performing.

[ReinhardKeil]

This problem is still present. The issue is can be duplicated using this project: https://github.com/Arm-Examples/Blinky_FRDM-K32L3A6.

The project contains a *.cbuild-set.yml that defines a context set to just - context: FRDM-K32L3A6.Debug+FRDM-K32L3A6.

Using cbuild Blinky.csolution.yml --context-set just builds the .Debug context - which is correct. However the *.cbuild-idx.yml refers also the .Release context.

build-idx:
  generated-by: csolution version 2.4.0
  description: Simple blinky example for FRDM-K32L3A6.
  csolution: Blinky.csolution.yml
  cprojects:
    - cproject: FRDM-K32L3A6.cproject.yml
  cbuilds:
    - cbuild: FRDM-K32L3A6.Debug+FRDM-K32L3A6.cbuild.yml
      project: FRDM-K32L3A6
      configuration: .Debug+FRDM-K32L3A6
    - cbuild: FRDM-K32L3A6.Release+FRDM-K32L3A6.cbuild.yml  # this file is not created
      project: FRDM-K32L3A6
      configuration: .Release+FRDM-K32L3A6

The file FRDM-K32L3A6.Release+FRDM-K32L3A6.cbuild.yml is not created and therefore missing.

I would recommend that *.cbuild-idx.yml only contains the files that are in the context-set.

[brondani]

The generation of *.cbuild-idx.yml with all contexts is currently needed for ensuring the correct generation of build files (CMakeLists). If one or more selected contexts depend on other contexts that are not selected at the same time, a partial *.cbuild-idx.yml would miss such dependencies leading to a fail.

I agree with Reinhard here:

I would recommend that *.cbuild-idx.yml only contains the files that are in the context-set.

But we should also improve or clarify the build approach without context-set which could be also relevant for the generation of cbuild-set.yml.

[ReinhardKeil]

I believe all you need is an error message, i.e.

error: context <project>.<build-type>+<target-type> not included in current build, output is required for context <project>.<build-type>+<target-type>

This would tell the user that there is a dependency between the projects and it is required to build the first context also.

[ReinhardKeil]

this is done