Ability to create standalone project - strong requirement.

[DavidJurajdaNXP]

Standalone project is self-contained project with all necessary files for build in single folder. This folder could be transported in form of archive and used in another machine.

[ReinhardKeil]

I believe we should analyze complex projects where the components are created from legacy code.

The main issue is how PLM would work on such projects. What should happen when a component is removed. Once the components are in the project tree the likelihood that they get modified by the user is higher. How do you envision the workflow for the user. Do you foresee that such "local" components get modified?

[tcsunhao]

I think there surely will be such issue that "The main issue is how PLM would work on such projects.". But for standalone project, it is just an option. Not a must use case but with this feature supported it provides with developers/amateurs a way/choice to quickly get a full-source-inside project to work/share.

[ReinhardKeil]

OK, any idea on how the workflow when editing the *.yml files could look like.

1. user enters: component: NXP::Driver:SPI into a *.cproject.yml. Maybe with an attribute "copy" or so.

What should happen next? Or should this be a command line option for "csolution" and related to all components in a project?

[LMESTM]

One question / discussion we've had in the forum before and that can be of importance: do you want the standalone project to contain only selected components or is that okay to have the complete pack ?

[tcsunhao]

Hi @LMESTM , here NXP Mcuxpresso way is that all contents of projects will be in a single pack with a reasonable folder structure defined by example owners. It is quite convenient locally here especially when we want to share example to debug issue or start something experimental. A package is usually big and inconvenient. BTW, can you send me the forum?

[tcsunhao]

Hi @ReinhardKeil, honestly, I am not quite deep into details of csolution yml at this time point. Attribute "copy" is one way. One thing I want to mention is that we need to keep a reasonable visible folder structure in any IDE workspace. In NXP way, it stands for both standalone/linked project. Although for most cases, the folder structure is just same as physical folder structure. Perhaps I can demo our NXP way more details in next meeting and then we can try to think about scenario for open CMSIS.

[ReinhardKeil]

Yep a demo next week would be great! Having a discussion may also help to come to a common solution. I think your biggest concern is the project tree in an IDE - is correct or is there more behind?

We should perhaps show how we solve this in VS Code (and Keil Studio) going forward. Would this be helpful too?

[tcsunhao]

Yes, data driven project tree is a very big concern/feature in NXP data model.

Your "show how we solve this in VS Code (and Keil Studio)" is perfect! Really looking forward to it!

[LMESTM]

Hi @LMESTM , here NXP Mcuxpresso way is that all contents of projects will be in a single pack with a reasonable folder structure defined by example owners. It is quite convenient locally here especially when we want to share example to debug issue or start something experimental. A package is usually big and inconvenient.

I think we have the same goal: we (ST) also want that any project is self-contained in a folder so that it can be zipped and shared easily by anyone. I'm just not sure to have the good understanding of what you refer to as pack or package (or component BTW) and whether one of those are CMSIS packs or not.

BTW, can you send me the forum?

I meant the discussion forum / meetings, where David is already involved. So I think @ReinhardKeil proposal to discuss next week is a good way forward.

[ReinhardKeil]

I think this topic intermixes two issues;

• PLM that requires somewhat self-contained projects can be regenerated easily.
• Access to source files from a project manager window.

Should we separate both topics as they are very different for complex projects?

[ReinhardKeil]

The picture below shows how IAR EW-Arm displays software components in the project tree.

The issue is that depending on the target-type and/or build-type the file list that are taken from components are different. Having everything copied into a file structure may therefore be just confusing when you work with multi-target configurations.

The *.CPRJ files contain all software packs along with version numbers that allows you to create one specify target-type/build-type. This might be where we aim for "self-contained" projects.

Let's discuss this in tomorrows meeting.

[tcsunhao]

Yes, Cclass, Cgroup, Csub can be used to construct project tree.

For a complex middleware component with lots files(head/source) located in different level folders, like The above IAR can well tell about the component, but still under Cclass/Cgroup/Csub for this component, there will be all files flat which have following defects:

1. It totally hide hierarchy organization information for the source of a component which may confuse users
2. It looks not tidy/nice.

The key is that from perspective of data model definition, Cclass/Cgroup/Csub is mainly used for classification of component while "project_path" serves source organization. They are not same concept.

And "project_path" is not only for source, but also for include, when we construct standalone project, if in the source, there is a relative include like "include ../../xx.h", we need to use include project_path as the include for the standalone project.

[tcsunhao]

I think such update is enough:

An optional attribute like "projectpath" or maybe other better naming can make it work.

[ReinhardKeil]

Thanks for your explanation. You envision that the usage would be for the NXP IDE - or wider then this?

[tcsunhao]

Now, NXP Mcuxpresso IDE/ Mcuxpresso config tool use "projectpath" to construct project tree and create standalone project. It is our very important feature.

One thing should be mentioned here is that from above attached IAR picture, it looks like different IDEs/tools may have different understanding/translation for the same attribute data(Keil IDE does different way in project tree with same data). It might be not good for end customer because for same component, in different IDEs, its layout is different.

Open CMSIS spec may need to give some standard or at least some recommendation for some data usage of pdsc.

For this specific topic, I think NXP project path way can be a candidate for project tree layout and standalone project implementation scenario.

[VGRSTM]

@tcsunhao from your experience here playing with project_path did you never face conflicts at a time ? I guess no worries if end user relies only on NXP packs I'm confident you're promoting sane metadata, but what about NXP & third parties mix ?

Extra question: If your current or envisioned Open-CMSIS based one, is end user getting a way to override per pack / component preset destination folder tree defaults ?

[tcsunhao]

Hi @VGRSTM, I would like to say that this "project_path" is an optional attribute, but a required source concept. For third parties which don't provide such "project_path" directly, we perhaps use physical path instead(Our tools are doing this way today with our manifest spec), we think default value for project_path is the physical path. So, there shouldn't be issue in construct project tree for any files without "project_path", anyway a source is with physical path. The basic feature of import/build component/examples from third parties are always met as they are today.

The "standalone project" is a advanced feature. If your component has relative include like include "xx/xx/xx/xx.h", you need to specifically provide "project_path" for your include corresponding to the source project_path, then you can enjoy standalone project with NXP IDE/tools of creating a standalone project, otherwise not.

From the above, all the packs can anyway work with basic features like import/build as they are today. If some packs want to enjoy more, they may(not always) need to provide some extra "project_path". It depends on IDE/tools dealing with data. We don't request other vendors to do the same way, but NXP IDE/tools need such basic concept.

About your extra question, I am not sure I understand it or not, in envisioned NXP open-CMSIS pack, user gets way to override per pack / component preset destination folder tree, they just need to edit "project_path" of the interested source in pdsc. (Precondition is open CMSIS agrees on this.)

(We expect everything can be data driven, with defaults.)

[VGRSTM]

@tcsunhao Thanks for comments.

About your extra question, I am not sure I understand it or not, in envisioned NXP open-CMSIS pack, user gets way to override per pack / component preset destination folder tree, they just need to edit "project_path" of the interested source in pdsc. (Precondition is open CMSIS agrees on this.)

Ok you have got my question right. My surprise here about your reply vs. what I've in mind is you're claiming end user will edit possibly pdsc file(s). According my view pdsc is golden ref. data being read only. So I was thinking to an extra process allowing end user to override (or just specify it if not part of pdsc) on the fly such project_path out of pdsc.

I support definitively all have to be data driven

Extra questions then please if I may:

Our tools are doing this way today with our manifest spec

So you today have a running mechanism. What's motivation to push part of your current manifest to pdsc Thanks project_path ? Is such manifest at pain maintenance or ? Myself I foresee value about such manifest on one side vs. pdsc which is maybe easier to revisit if required than having to revisit collection of packs embedding pdscs. Plus external manifest is allowing you to possibly handle third parties who may have not contributed to project_path

For third parties which don't provide such "project_path" directly, we perhaps use physical path instead

Not fully clear to me. Do you means if third party doesn't contribute to project_path then you promote a default local repository folder tree which may be <project_root>/Packs/<vendor>/<pack name>/.... or just a copy of pack itself file structure

If your component has relative include like include "xx/xx/xx/xx.h", you need to specifically provide "project_path" for your include corresponding to the source project_path

Not fully clear to me. Do you means locally as part of your project context you are updating component golden files on the fly (at local copy time) remapping some include path inside to stick to local file structure which may have been "fully jeopardized" thanks project_path usage ?

Sorry in advance if some weird questions but I guess we need clear and full understanding of your target & related constraints having a chance to get an agreement here. I support myself definitively the standalone / self contain project concept.

[tcsunhao]

@VGRSTM , First, I don't think I get your question right "user gets way to override " because as you said and we all know pdsc file is read-only with a lock in left tap. Edit file is just a workaroud to try something locally, not an regular way. About "extra process allowing end user to override", it sounds nice although NXP tool doesn't do it Today.

Yes, we have a running mechanism. And we want to better our usage of open CMSIS because we realize it is very popular and generic, and we decide to totally support/consume the pdsc data(we don't request other IDEs to do the same, we need data), so that more wide world can use NXP IDE/tool. Manifest will still be supported in NXP.

"For third parties which don't provide such "project_path" directly, we perhaps use physical path instead" Let's take above as an example, if no "project_path", we will use "devices/MK64F12/drivers" to construct project tree for each source. We think "Packs/vendor/pack name" is used to classify component, under this path, lots of sources(especially for complex component) will still be flat, not well organized. We also realize that our way may lose component classification information. We will try to provide such information somehow. But this is related to IDE behavior on data, not data itself.

"If your component has relative include like include "xx/xx/xx/xx.h", you need to specifically provide "project_path" for your include corresponding to the source project_path" It is related to NXP way to achieve standalone project. When we create standalone project, we may remap project if you provide different project_path to physical path. If source remapped, includes should be remapped correspondingly. If you put gpio.h into "drivers" from devices/MK64F12/drivers, you should also update the include to "drivers" from "devices/MK64F12/drivers". I am curious how ST do this.

Thanks.

[VGRSTM]

@tcsunhao

About "extra process allowing end user to override", it sounds nice although NXP tool doesn't do it today

I've got question because per #85 (if extended to component we may think same) we may open door to end user control on local destination. In my view such yml based could override a possible pdsc project_path attribute. Was curious if such capability already running on your side or envisioned. I understand your current manifest is not under user control.

I am curious how ST do this.

At that time we are not revising at all pack / component content doing local copy we are keeping original structure. Rational is it ease our life and as a pack / component maybe a third party one we are not taking under our own to "hack" it. Cons. is end user may end up with complex folder structure. I understand on your side Thanks current manifest you're supporting optional project_path based remapping only if NXP contributions keeping third party as defined. Pushing for project_path being part of pdsc std definition you want open door to third parties taking care themselves to promote local remapping capability of their own contribution if they want.

[DavidJurajdaNXP]

What's motivation to push part of your current manifest to pdsc?

We believe there is need for standardized, open software exchange format. We are considering replacement of our manifest by such standardized format in future, but we would like to have support for features like project_path in the standard.

Our tools are using data driven project explorer (tree) structure. In order to satisfy this, we need to deliver expected data. project_path = data driven project explorer view

[ReinhardKeil]

I think we can support a feature like project_path but leave it actually implementation defined if an IDE/tool wants to support it or not. Would this already solve your problem?

[tcsunhao]

I think we can support a feature like project_path but leave it actually implementation defined if an IDE/tool wants to support it or not. Would this already solve your problem?

Yes, I think this way solve our problem.

[DavidJurajdaNXP]

One of the expectations related to standalone project is minimal size of such project. In relation to this, I would like to mention issue with presence of examples in packs. If we design standalone project in way, where we would copy whole content of pack into project folder (or simply place installation folder into project folder), there will be significant overhead because of examples which are inseparable once described in pack. May we deliver examples in example specific pack - separated from pack with software components?

[LMESTM]

@DavidJurajdaNXP I fully support your point. To achieve reasonable pack size, examples should be delivered separately. We need packs with reasonable content scope to limit size and achieve possible use in local copy / standalone project.

[ReinhardKeil]

Completely agree with your view. We started with Packs in the current format to allow packing of SDKs that included also examples. Remember this was ~10 years ago.

Today with GitHub there are be better ways to deliver examples. So we would need to work on guidelines for packs going forward.

[ReinhardKeil]

Another way to improve situation with current packs might be a Pack Copy utility that excludes examples, documentation, etc. So just copies the software components.

[jkrech]

I would prefer to rethink the granularity of packs being provided rather than creating "partial" copies of packs. One option to deliver examples could also be based on public repositories. In that case the pdsc example description only contains the link to the respective repository. We could even consider having a single repository containing multiple "related" examples.

[ilg-ul]

Source xPacks, which use the same format as npm packages, use a .npmignore file to list all files/folders that are not to be packed in the published archive.

Thus, the project repo can contain tests, examples, documentation, etc as usual, but the released package retains only the essential content.

A good example of such a project is

• https://github.com/xpack-3rd-party/arm-cmsis-core-m-xpack

which, although it contains a full fork of the original CMSIS git, in the end packs only the CMSIS/Core/Include folder, the CMake and meson configurations and ignores everything else:

 @xpack-3rd-party/arm-cmsis-core-m@5.4.0-5 pack
> npm pack

npm notice 
npm notice 📦  @xpack-3rd-party/arm-cmsis-core-m@5.4.0-5
npm notice === Tarball Contents === 
npm notice 615B    CHANGELOG-XPACK.md                     
npm notice 3.1kB   CMakeLists.txt                         
npm notice 12.1kB  CMSIS/Core/Include/cachel1_armv7.h     
npm notice 28.0kB  CMSIS/Core/Include/cmsis_armcc.h       
npm notice 56.0kB  CMSIS/Core/Include/cmsis_armclang_ltm.h
npm notice 48.4kB  CMSIS/Core/Include/cmsis_armclang.h    
npm notice 9.5kB   CMSIS/Core/Include/cmsis_compiler.h    
npm notice 63.4kB  CMSIS/Core/Include/cmsis_gcc.h         
npm notice 29.1kB  CMSIS/Core/Include/cmsis_iccarm.h      
npm notice 1.7kB   CMSIS/Core/Include/cmsis_version.h     
npm notice 115.0kB CMSIS/Core/Include/core_armv8mbl.h     
npm notice 182.7kB CMSIS/Core/Include/core_armv8mml.h     
npm notice 276.2kB CMSIS/Core/Include/core_armv81mml.h    
npm notice 41.6kB  CMSIS/Core/Include/core_cm0.h          
npm notice 49.9kB  CMSIS/Core/Include/core_cm0plus.h      
npm notice 42.6kB  CMSIS/Core/Include/core_cm1.h          
npm notice 109.5kB CMSIS/Core/Include/core_cm3.h          
npm notice 121.0kB CMSIS/Core/Include/core_cm4.h          
npm notice 138.9kB CMSIS/Core/Include/core_cm7.h          
npm notice 121.6kB CMSIS/Core/Include/core_cm23.h         
npm notice 189.0kB CMSIS/Core/Include/core_cm33.h         
npm notice 189.0kB CMSIS/Core/Include/core_cm35p.h        
npm notice 280.3kB CMSIS/Core/Include/core_cm55.h         
npm notice 46.6kB  CMSIS/Core/Include/core_sc000.h        
npm notice 108.5kB CMSIS/Core/Include/core_sc300.h        
npm notice 11.7kB  CMSIS/Core/Include/mpu_armv7.h         
npm notice 11.4kB  CMSIS/Core/Include/mpu_armv8.h         
npm notice 22.8kB  CMSIS/Core/Include/pmu_armv8.h         
npm notice 2.7kB   CMSIS/Core/Include/tz_context.h        
npm notice 1.1kB   LICENSE-XPACK                          
npm notice 11.4kB  LICENSE.txt                            
npm notice 15.3kB  README.md                              
npm notice 1.6kB   meson.build                            
npm notice 1.3kB   package.json                           
npm notice === Tarball Details === 
npm notice name:          @xpack-3rd-party/arm-cmsis-core-m            
npm notice version:       5.4.0-5                                      
npm notice filename:      @xpack-3rd-party/arm-cmsis-core-m-5.4.0-5.tgz
npm notice package size:  328.6 kB                                     
npm notice unpacked size: 2.3 MB                                       
npm notice shasum:        f5847fac4d7229d6855fac75bf865e9035fe951e     
npm notice integrity:     sha512-67+oGXTD3qw6R[...]cY09BDAHIOikg==     
npm notice total files:   34                                           
npm notice 
xpack-3rd-party-arm-cmsis-core-m-5.4.0-5.tgz

This makes using the CMSIS Core headers quite convenient, compared with using the original CMSIS Pack, which is huge. This is even more convenient in CI environments, which reconstruct the project from components at each run.

[ilg-ul]

I would prefer to rethink the granularity of packs being provided

Normally, from the dependency management point of view, the granularity of packs should be one or more files that share a common version (probably a component in your terminology).

Apart from a common version, the rules to group files together are also simple and should aim for standalone packages, for example files that never participate in a build together should be placed is separate packages, files that always participate in a build together should be placed in the same package, etc.

[fred-r]

I would prefer to rethink the granularity of packs being provided rather than creating "partial" copies of packs. One option to deliver examples could also be based on public repositories. In that case the pdsc example description only contains the link to the respective repository. We could even consider having a single repository containing multiple "related" examples.

Let's also keep in mind that there is a use-case to have signed packs. Signature validity must be verifiable at any point in time of the pack usage (installation, dependencies solving, build). So, if we start copying parts of packs then we lose the capability to check the signature.

My proposal is that at least signed packs must not be split.