Reducing CMSIS Packs size

[rapgenic]

My problems:

• I build docker images with tools for developing for microcontrollers (which contain toolchains, debuggers, utilities, etc.). In this image I ship pyOCD as well, and as one requisite for these images is not connecting to the internet, I need to pre-download CMSIS packs for the targets that I want to use. CMSIS Packs are a lot of megabytes, thus the docker images grow of gigabytes in size with just a few packs!
• CMSIS Packs waste a lot of space even in my personal computer

Proposed solution

Looking at the code, it seems to me that of a CMSIS Pack only the following files are used:

• the .pdsc file
• the FlashAlgo files
• the .svd files
• more?

In any case the impression is that a lot of files in the pack are completely useless for pyocd.

What I'm proposing is that we find some way to store on the disk only the relevant parts of the pack, which is really small compared to the full size file. This would allow to save a very large amount of space.

Possible implementations

I have thought of two ways this could be implemented, which I'm giving just as a pointer of course:

• Download the zip file and then delete from the local copy the files that are not needed.
  • PROS:
    • Simple to implement, just delete files from zip after downloading
    • Could be done by pyOCD itself
  • CONS:
    • Might be forward incompatible, if in future versions of pyOCD some other files are needed, there may be errors and packs would need to be downloaded again.
    • The whole pack needs to be downloaded anyway
• Access only the needed part of the zip pack directly from the server via HTTP and extract on the fly the needed files (see stackoverflow or this library)
  • PROS:
    • No need to download the full pack
  • CONS:
    • Probably more tricky to implement (as it would be implemented in cmsis-pack-manager I suppose)
    • HTTP servers might not like GET requests for only a part of the file (only maybe, as it should be standard HTTP anyway)

Sorry for the long post, I just wanted to start a conversation and give a few ideas on this, given that in my opinion this is one of the few downsides of pyOCD compared for example to openOCD.

[flit]

Hi @rapgenic 👋🏽

Yes, some of the packs, especially the STM32 packs, are quite large. You aren't the only one that's run into this. For Mbed Studio we stripped nonessential files from the included packs. And there's at least one other user that is building containers that include packs.

You're right that the required content is the .pdsc, .svd, and .FLM files. Potentially also .dbgconf and .sdf files with more debug info, but those won't be used for the foreseeable future, and the latter is very rare, in any case. There shouldn't be an issue with needing additional files because the packs would have to be updated, and therefore re-downloaded, to reference any new files defined in an updated Open-CMSIS-Pack spec.

First, as long as pyOCD is using cmsis-pack-manager (CPM) to handle packs it's difficult to change things in the short term. That's because CPM is (supposed to be, at least) a general pack management solution. That said, the only users I know of are pyOCD and probe-rs, and they only use the device description data. Still, it would be a major change. (Open-CMSIS-Pack now has a pack manager tool for software packs, so the possibility of using CPM for this is almost nonexistent.)

I have been thinking about replacing CPM with a pure-Python pack manager more geared to pyOCD. Having CPM be mostly written in Rust is kind of a pain for build and distribution reasons (also for me, since I haven't had time to learn Rust yet! ☹️). However, this is a pretty big undertaking, and there are a lot of competing priorities that are probably more important for most users.

Access only the needed part of the zip pack directly from the server via HTTP and extract on the fly the needed files

Sounds like this wouldn't work for your requirement of supporting no network connectivity?

These are the options as I see it:

• Since the primary use case for reducing pack size seems to be for building containers or other distributions, the easiest approach for the immediate future might simply be a script that strips packs prior or during your container build process. We could build a reusable script that is included in either the CPM or pyOCD repos.
• Adding an option to CPM to strip packs after download, exposed by pyocd pack install.
• Replace CPM with pure-Python pack manager, also with a pack stripping feature.

[rapgenic]

Hi @flit, sorry, I seem to miss quite a lot of notifications recently!

also for me, since I haven't had time to learn Rust yet!

Me neither, I've tried many times with no success, not having a real project that needed it...

However, this is a pretty big undertaking, and there are a lot of competing priorities that are probably more important for most users.

I totally understand this, in fact I tend too towards simpler solutions

Sounds like this wouldn't work for your requirement of supporting no network connectivity?

Yes, that was a bit off topic, but I was thinking of the larger picture:

• On one hand that feature (partial download from online packs) would reduce the network traffic for any user, and the download time as a consequence.
• On the other we have noticed that servers that host cmsis packs tend to be quite restrictive with rate limiting. I was wondering if, while building my container, downloading GBytes of packs would be tolerated or if suddenly I wouldn't be able to download packs anymore due to the excessive amount of traffic.

---

Regarding the three options you presented, I quite agree with you, modifying the cmsis-pack-manager or even rewriting one for pyocd, might be The Right Way™, but it's quite complicated and time consuming.

A script that strips packs is surely feasible and I was about to write one myself before opening this issue. It could even be exposed to the cli like pyocd pack compress or something like that.

In addition to those I'd like to propose one other idea I had, which is a middle ground between working on the CPM and using a custom script.

I suppose there is a common index where all pack information and download location are published. We could create a timed workflow (e.g. github action) triggered periodically that would fetch pack information, determine which packs have been updated (probably using CPM itself) and then create and upload a pypi package for each cmsis pack, containing only the stripped version (which would be a few of MB maximum, so not too big).

Cmsis packs could then be installed as pypi packages, already stripped, and without needing to rely on external servers (which to be fair have had a few hiccups in the past), we would even have for free some form of "pack management" from the pypi itself.

The only modification needed for pyocd, then, would be adding a way only to detect and use the packs installed from the pypi, without touching the already existing infrastructure.

PROS:

• Minimal intervention in existing codebase
• Almost automatic, ideally set and forget, method
• No need to download directly from vendor servers + packs always available for download
• History of packs is preserved (this is an issue we haven't talked about, so I'm just quickly mentioning it, but for full reproducibility, I'd like to be able to have always the same cmsis packs, which is not possible ATM as CPM always downloads the latest)
• Already stripped packs

CONS:

• The rate limiting problem is just shifted to the workflow runner (which still has to download GBs of data)
• A second parallel packaging method might be a little confusing (but no need to advertise it from the beginning)
• A bit more complicated than a simple script

[rapgenic]

Hi, I had some time today, so I started putting up a prototype of my last proposal.

At the moment I focused on an easy way of repackaging cmsis packs into python wheels, see https://github.com/protech-engineering/pyocd-cmsis-pack if you're interested (the code is ugly, but it's a starting point).

Quite ironically the only thing missing is the stripping of pack files, because there seems to be no easy way to do this using python standard library :laughing:

Most details are in the README and the code is quite small, so I'm not explaining everything here again...

[flit]

Hi @rapgenic,

That's a pretty interesting idea that I wouldn't have thought of in the same way! I'm fine with supporting it in pyocd. 😄

The main concern from my side is the addition of another target support source. However, for a while I've wanted to have target support provider plugins. This can be useful for proprietary or in-development targets, or customizing existing targets.

There are at least couple possible ways to integrate into pyocd:

1. Modify the code in pyocd.target.pack.pack_target to support pack plugins. Easier, smaller change but limited to CMSIS-Packs.
2. Full target support provider plugins. This would effectively replace the code in pyocd.board.board.Board.__init__() with code that asks plugins if they can support the target.
   • This has implications in several places in the code, mostly for things like listing and looking up supported targets.
   • Pack plugins could be based on common code to implement a prepackaged-pack-based target provider plugin. The main DFP handling code would remain in pyocd.

Also, the above two options could both be implemented.

Well, I could could go on for a long time here… This is probably enough to start the discussion going, though.

Btw, I should mention the cmsis-pack-monitor script I wrote a while ago. It has a fairly complete pack index and .pdsc download implementation, though it doesn't download the full packs.

[rapgenic]

Hi @flit,

Sorry for the long delay, I had a busy week...

That's a pretty interesting idea that I wouldn't have thought of in the same way! I'm fine with supporting it in pyocd.

That's great!

Regarding the ways to integrate is in pyocd, I definitely think that target support plugins are very interesting, however it seems to me that their purpose might be a little different to what we want to do with cmsis packs.

From what you said before, I understand that target plugins would support directly the addition of new targets, using pyOCD classes like Board, which are already very specific (read it high level if you prefer). What I mean is that I'd expect that a plugin that encapsulates a pack would need to parse the pack in some way and provide higher level objects (like Board and similar) already prebuilt from the pack to pyocd api.

The advantage of pack plugins, instead, in my opinion, is that we can provide directly the CmsisPack object (or a pack path, if you prefer, which would be even better) and then pyocd would reuse the existing structure to extract each target and all of their details.

Unless of course a target plugin allowed to pass a cmsis pack path to pyocd, which however would be like joining the two approaches, as you said.

In the end what I'm a little worried about is that we make a too complicated system to support a use case which is for now very simple, which is just adding a cmsis pack in a different way than before. Target plugins are interesting, but may be too generic for this purpose.

Btw, I should mention the cmsis-pack-monitor script I wrote a while ago. It has a fairly complete pack index and .pdsc download implementation, though it doesn't download the full packs.

This is really interesting thanks! I'll definitely have a look, even though CPM itself might be enough, from the demo I did a few days ago it seems to me that it does its job quite well!

[rapgenic]

Hi, I had to postpone this development due to other priorities at the moment... I just want to share the oneliner that I've been using to strip the packs from all the unnecessary files in my dockerfiles for the moment, in case anyone is facing the same need:

cd ~/.local/share/cmsis-pack-manager/
for file in $(find . -iname "*.pack"); do zipinfo -1 $file | sed '/^.*\(flm\|pdsc\|svd\|dbgconf\|sdf\)/Id' | xargs zip -d $file; done

Sometimes I'm amazed how easy it is to do things with few characters of bash scripting!

Of course it's not the cleanest solution, but for now it does what I need