[csolution] [cbuild] directory tree gets difficult to use

fred-r commented 2 years ago

Let's consider a csolution with 1 board and 3 compilers:

$ tree
.
|-- README_OpenCMSIS_Project.md
|-- gpio-toggle.cproject.yml
`-- gpio-toggle.csolution.yml

0 directories, 3 files

Then I create the cprjs:

$ csolution -s gpio-toggle.csolution.yml convert
C:/Cube/stm32cube_fw_all/examples/hal/gpio/toggle/projects/gpio-toggle.AC6+B-U585I-IOT02A.cprj - info csolution: file generated successfully
C:/Cube/stm32cube_fw_all/examples/hal/gpio/toggle/projects/gpio-toggle.GCC+B-U585I-IOT02A.cprj - info csolution: file generated successfully
C:/Cube/stm32cube_fw_all/examples/hal/gpio/toggle/projects/gpio-toggle.IAR+B-U585I-IOT02A.cprj - info csolution: file generated successfully

$ tree
.
|-- README_OpenCMSIS_Project.md
|-- RTE
|   `-- Device
|       `-- STM32U585ZITx
|           |-- STM32U585xx_FLASH.ld
|           |-- stm32_external_env.h
|           |-- stm32_external_env.h@0.0.1
|           |-- stm32u585xx_flash.icf
|           |-- stm32u585xx_flash.sct
|           |-- system_stm32u5xx.c
|           `-- system_stm32u5xx.c@0.0.1
|-- gpio-toggle.AC6+B-U585I-IOT02A.cprj
|-- gpio-toggle.GCC+B-U585I-IOT02A.cprj
|-- gpio-toggle.IAR+B-U585I-IOT02A.cprj
|-- gpio-toggle.cproject.yml
`-- gpio-toggle.csolution.yml

3 directories, 13 files

Already at this stage it becomes pretty fuzzy with 1 RTE with 1 device but 3 CPRJs at the same level.

Now, if I build:

$ cbuild.sh gpio-toggle.AC6+B-U585I-IOT02A.cprj

$ tree
.
|-- README_OpenCMSIS_Project.md
|-- RTE
|   |-- Device
|   |   `-- STM32U585ZITx
|   |       |-- STM32U585xx_FLASH.ld
|   |       |-- stm32_external_env.h
|   |       |-- stm32u585xx_flash.icf
|   |       |-- stm32u585xx_flash.sct
|   |       `-- system_stm32u5xx.c
|   `-- _gpio-toggle.AC6_B-U585I-IOT02A
|       `-- RTE_Components.h
|-- gpio-toggle.AC6+B-U585I-IOT02A.cprj
|-- gpio-toggle.AC6+B-U585I-IOT02A_IntDir
|-- gpio-toggle.AC6+B-U585I-IOT02A_OutDir
|-- gpio-toggle.GCC+B-U585I-IOT02A.cprj
|-- gpio-toggle.IAR+B-U585I-IOT02A.cprj
|-- gpio-toggle.cproject.yml
`-- gpio-toggle.csolution.yml

36 directories, 42 files

Now we end up with many folders all at the same level....

Wouldn't it make sense to have something organized:

at a first level by target-type
at a second level by build-type to have something more organized ?

brondani commented 2 years ago

Good suggestion, let's work on that.

VGRSTM commented 2 years ago

Already I would push for some good practice having one repository folder per cproject.yml file. Sounds to me csolution / cproject / clayer is promoting hierarchical structure.

Based on your trial @fred-r I'm just realizing if multi target toolchains we are ending with one cprj per toolchain. @brondani no more intent to end with a single cprj file despite toolchain variations ? ... cprj schema is allowing such. Not pushing strongly for just for my information.

I'm in sync. with @fred-r we have to organize a bit all such. Possibly ?: In any case we have to keep path in general as short a possible thinking some Windows OS & possibly toolchains long path limitation.

Note: If single cprj per target device

brondani commented 2 years ago

@VGRSTM as discussed in the last Open-CMSIS-Pack meeting I'm writing here my considerations:

For clarity and simplicity we should narrow the CPRJ schema and accept only a single toolchain.
Concerning the CPRJ and RTE config files location we need to consider the user may require to store it alongside the csolution/cproject files ("on-source") or in a separated folder ("out-of-source").
RTE config files for a given context need to be in a common RTE folder. For this reason currently a solution with layers and RTE files require an "out-of-source" approach. Are there alternatives?

VGRSTM commented 2 years ago

Considering @brondani valuable comments and current Open-CMSIS trend here is a new structure proposal:

CONFIG_ID="./\<customConfigPathPattern>" // "\<target-type>\/\<build-type>" as default ? RTEDIR="./\<customRteDir>" // "./RTE" as default ? BUILDDIR="./\<customBuildDir>/" // "./build/$CONFIG_ID" as default CPRJDIR="./\<customLockDir>/" // "./.lock" as default ?

./AppProject/AppProject.csolution.yml

./AppProject/Project_1/Project_1.cproject.yml ./AppProject/Project_1/$CPRJDIR/Project_1.\<target-type>+\<build-type>.cprj ./AppProject/Project_1/$BUILDDIR/inDir/CMakeLists.txt ./AppProject/Project_1/$BUILDDIR/outDir/src/main.o ./AppProject/Project_1/$BUILDDIR/outDir/Project_1.elf ./AppProject/Project_1/$RTEDIR/linkerscript.ld

./AppProject/Project_2/Project_2.cproject.yml ./AppProject/Project_2/$CPRJDIR/Project_2.\<target-type>+\<build-type>.cprj ./AppProject/Project_2/$BUILDDIR/inDir/CMakeLists.txt ./AppProject/Project_2/$BUILDDIR/outDir/src/main.o ./AppProject/Project_2/$BUILDDIR/outDir/Project_2.elf ./AppProject/Project_2/$RTEDIR/linkerscript.ld

High level message is "please hard code nothing but promote variables". Open-CMSIS is open solution we have to keep it open to integrator / end user choices.

Some rational then about proposal:

cprj file(s) location
- *.cprj files mean if PLM so have to be part of source controlled files. I assume here .cprj is moving to act as a lock file ensuring build reproducibility (ACI, project sharing from one contributor to another, ...).
- Due to possible combinatory (target & build types) we are going to end up with multiple .cprj files
- I want to prevent basic end user to be exposed to too much metadata files and in any case we have to move metadata out of end user sources. If project description, .yml files are root of trust so first end user concerns no trouble here to get them at related project root. .cprj files mean but I assume tools integrating Open-CMSIS proposal will promote automation on top of and only advanced end users may look at. => Sounds required to get dedicated repository dedicated to *.cprj files
build artifacts repository
- I want to get capability to add basic .gitignore to project repository => Sounds required to get dedicated repository dedicated to build artifacts files
RTE repository
- See "RTE" directory configurable

fred-r commented 2 years ago

High level message is "please hard code nothing but promote variables".

Yes, I fully agree and I would even say, let's not assume that everything goes under a top level directory. We may have some organisations with:

projects on one hand
target code on the other hand at the same level

Also, the tooling should not assume it is "alone" in the tree. By this I mean that the end-user may create extra folders on his own. So, when the tool is doing clean-up or parsing, these folders must be ignored if needed.

VGRSTM commented 2 years ago

@brondani Thanks past weekly call review I've well noticed comment about possible data clash(es) because flexible structure first guess proposal. Your comment is fully valid, Thanks ! Yes if considering case we are facing "flat" usage (no Project 1 & Project2 file system folders, means all yml files sharing same location) we have conflicts if build artefacts. Let have a refined proposal then !

My first guess is lacking so about "qualifier" uniqueness. I'm thinking way to solve is to add some $PROJECT support reflecting project name (which is *.cproject.yml files prefix ? TBC ... see [projmgr] Get rid of project name usage as much as possible. cprj file(s) out of cproject.yml file(s) are already made unique Thanks to project name usage under the hood .. let's allow same based on private variable introduction ("private" means devtools owned, having value assigned at runtime by devtools processing). Is such sounds better to you ? :

CONFIG_ID="./\<customConfigPathPattern>" // "\<target-type>/\<build-type>" as default ? RTEDIR="./\<customRteDir>" // "./RTE" as default ? BUILDDIR="./\<customBuildDir>/" // "./build/$PROJECT/$CONFIG_ID" as default CPRJDIR="./\<customLockDir>/" // "./.lock" as default ?

./AppProject/AppProject.csolution.yml

./AppProject/Project_1/Project_1.cproject.yml ./AppProject/Project_1/$CPRJDIR/Project_1.+.cprj ./AppProject/Project_1/$BUILDDIR/inDir/CMakeLists.txt ./AppProject/Project_1/$BUILDDIR/outDir/src/main.o ./AppProject/Project_1/$BUILDDIR/outDir/Project_1.elf ./AppProject/Project_1/$RTEDIR/linkerscript.ld

./AppProject/Project_2/Project_2.cproject.yml ./AppProject/Project_2/$CPRJDIR/Project_2.+.cprj ./AppProject/Project_2/$BUILDDIR/inDir/CMakeLists.txt ./AppProject/Project_2/$BUILDDIR/outDir/src/main.o ./AppProject/Project_2/$BUILDDIR/outDir/Project_2.elf ./AppProject/Project_2/$RTEDIR/linkerscript.ld

VGRSTM commented 2 years ago

Same way I've introduced $PROJECT private variable sounds to me good to add some extra ones:

$BUILD_TYPE
$TARGET_TYPE
$COMPILER Which may serve the same

VGRSTM commented 2 years ago

PR #266 proposal aiming to update reference schemas opening door to dedicated output directories support if devtools

fred-r commented 2 years ago

In your proposal Vincent, can we accept ../ to go up in the tree ?

VGRSTM commented 2 years ago

@fred-r I guess yes ../ can be supported. As far of today already nothing prevent to do such. According to me has to be supported as part of standard and related devtools. Then up to end user to define its own way of working with. If we expect to get a project (cproject.yml) being "shared" / "reused" by multiple applications (csolution.yml) sounds to me ../ is must have.

fred-r commented 2 years ago

Another problem in our naming today is the length of the paths it generates.

It is easy to hit this limit: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation

|-- test.AC6_UART+B-U585I-IOT02A_IntDir
|   |-- CMakeCache.txt
|   |-- CMakeFiles
|   |   |-- 3.22.1
|   |   |   |                       `-- CMakeCCompiler.cmake
|   |   |                           `-- CMakeSystem.cmake
|   |   |-- CMakeOutput.log
|   |   |-- TargetDirectories.txt
|   |   |-- cmake.check_cache
|   |   |-- rules.ninja
|   |   `-- test.AC6_UART+B-U585I-IOT02A.dir
|   |       |                       `-- C_
|   |       |   |                   `-- CMSIS_PACKS
|   |       |   |                   `-- STMicroelectronics
|   |       |   |       |           `-- STM32U5xx_DFP
|   |       |   |       |           `-- 0.1.1
|   |       |   |       |           `-- Source
|   |       |   |       |           `-- startup_stm32u585xx.o
|   |       |   |       `-- STM32U5xx_Drivers
|   |       |   |           `-- 0.1.1
|   |       |   |               `-- Src
|   |       |   |                   |-- stm32u5xx_hal.o
|   |       |   |                   |-- stm32u5xx_hal_cortex.o
|   |       |   |                   |-- stm32u5xx_hal_dma.o
|   |       |   |                   |-- stm32u5xx_hal_dma_ex.o
|   |       |   |                   |-- stm32u5xx_hal_gpio.o
|   |       |   |                   |-- stm32u5xx_hal_pwr.o
|   |       |   |                   |-- stm32u5xx_hal_pwr_ex.o
|   |       |   |                   |-- stm32u5xx_hal_rcc.o
|   |       |   |                   |-- stm32u5xx_hal_rcc_ex.o
|   |       |   |                   |-- stm32u5xx_hal_uart.o
|   |       |   |                   `-- stm32u5xx_hal_uart_ex.o
|   |       |   `-- Cube
|   |       |       `-- background
|   |       |           `-- embsw_archi_studies
|   |       |               `-- simple_trace
|   |       |                   |-- Src
|   |       |                   |   `-- stm32_simple_trace_UART.o
|   |       |                   `-- internal
|   |       |                       |-- example_status.o
|   |       |                       |-- main.o
|   |       |                       |-- targets
|   |       |                       |   `-- B-U585I-IOT02A
|   |       |                       |       |-- stm32_system.o
|   |       |                       |       `-- stm32_usart1.o
|   |       |                       `-- test.o
|   |       `-- c4672b92c63e51bfab1fbcdac4d19a9b
|   |           `-- simple_trace
|   |               `-- internal
|   |                   `-- RTE
|   |                       `-- Device
|   |                           `-- STM32U585ZITx
|   |                               `-- system_stm32u5xx.o
|   |-- CMakeLists.txt
|   |-- build.ninja
|   |-- cmake_install.cmake
|   `-- compile_commands.json

We can see for instance:

SKIPPED then "test.GCC_UART+B-U585I-IOT02A_IntDir/CMakeFiles/test.GCC_UART+B-U585I-IOT02A.dir/c4672b92c63e51bfab1fbcdac4d19a9b/simple_trace/internal/RTE/Device/STM32U585ZITx/system_stm32u5xx.c.obj"

==> already 182 chars...

jankap commented 2 years ago

Another problem in our naming today is the length of the paths it generates.

It is easy to hit this limit: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation

I'm running into this issue easily, too. Especially when CMSIS Packs are stored in %appdata% which is resolved to very long intermediate paths under the cmake build folder.

ReinhardKeil commented 2 years ago

Take a look to my proposal here #314

ReinhardKeil commented 2 years ago

There is not activity on that since a while. What is the take on my proposal #314

ReinhardKeil commented 1 year ago

Closing this issue as it is somewhat solved

Open-CMSIS-Pack / devtools

[csolution] [cbuild] directory tree gets difficult to use #221