schemas: Add a schema for data layout in firmware

[sjg20]

I am adding a PR since it seems that the reviewers are happy, or at least silent

[sjg20]

Hi Rob, what should I do here?

[robherring]

I said at the beginning that /firmware is already in use for providers (e.g. clock or power controllers, etc.) which happen to be implemented by runtime firmware rather than in h/w. This use is something different.

What is the need for this to live in dtschema? Who's using it?

[sjg20]

This version has been modified to be more generic based on comments on v1.

The concept is described here:

https://u-boot.readthedocs.io/en/latest/develop/package/binman.html#image-description-format

My goal is to describe the layout of the firmware (attached to the device that holds it) so that it can allow firmware update on any supported board:

https://github.com/fwupd/fwupd/blob/main/plugins/vbe/README.md#board-information

I am not wedded to using the 'firmware' node if there is a better one.

Really this is a generalisation of this:

https://www.kernel.org/doc/Documentation/devicetree/bindings/mtd/partitions/

so could perhaps go in there, if you prefer.

[robherring]

We certainly shouldn't duplicate the existing partitions bindings. What's missing from them (I assume we're mostly talking about "fixed-partitions" which has been around forever I think (before me))?

[sjg20]

Quite a few things, for example:

• indication of what format/type/thing is in each partition
• hierarchy within the firmware (sections within sections)
• compression
• hashing / security

I remember actually trying to do something with fixed-partitions perhaps 10 years ago...but I know a lot more now

[robherring]

Quite a few things, for example:

* indication of what format/type/thing is in each partition

That's generally "compatible"

* hierarchy within the firmware (sections within sections)

You can have partitions within partitions.

* compression

* hashing / security

Seems like things the OS would want to know too.

I'm surprised A/B image handling is not on your list.

[sjg20]

OK, so are you saying that I should bring in partition.yaml, partitions.yaml and fixed-partitions.yaml from Linux and then build on the latter?

Re hierarchy I tried to do this in ChromeOS years ago but at the time it wasn't allowed. But I see it changed some 7 years later:

d2ad00eb78792 dt-bindings: mtd: explicitly document nesting partitions descriptions

Re A/B update...it was just a small selection of what is needed.

[robherring]

OK, so are you saying that I should bring in partition.yaml, partitions.yaml and fixed-partitions.yaml from Linux and then build on the latter?

Please extend it in place. First, every time we copy something over, I need to delete it from the Linux tree (the duplicate causes a warning) and bump the dtschema version required. Also, I think we'd have to move nvmem bindings over or fix how they are included. Second and more importantly, I want MTD maintainers to review any changes. If we move it out, then they won't.

Also, the structure of the schemas needs some rework as we don't have unevaluated properties checking everywhere. It's on my list to look at.

[sjg20]

Rob, my concern is that when I send a firmware-related change to Linux, I will spend the next year of my life trying to explain why it cannot be done in userspace, why anyone would care about this binding, who uses it, etc.

Can you point me to DT features added to Linux recently that are not used by Linux?

[robherring]

Rob, my concern is that when I send a firmware-related change to Linux, I will spend the next year of my life trying to explain why it cannot be done in userspace, why anyone would care about this binding, who uses it, etc.

For bindings, "done in userspace" is usually a question of who wants to change some setting and when. If it changes one user to the next or at runtime, then it shouldn't be in DT. Don't think that's going to apply here. More likely is the "where is the user" question, and I'm more than happy to have that argument.

Can you point me to DT features added to Linux recently that are not used by Linux?

bootph-*

Schema wise, not sure. I took Mali GPU binding before there was any upstream driver, but that was sometime back now. Frankly, the issue rarely comes up.

[sjg20]

Rob, my concern is that when I send a firmware-related change to Linux, I will spend the next year of my life trying to explain why it cannot be done in userspace, why anyone would care about this binding, who uses it, etc.

For bindings, "done in userspace" is usually a question of who wants to change some setting and when. If it changes one user to the next or at runtime, then it shouldn't be in DT. Don't think that's going to apply here. More likely is the "where is the user" question, and I'm more than happy to have that argument.

There are so many threads with Linux people arguing against firmware using DT that I have lost track. We have senior people at Linaro on the record, we have distro maintainers, just for a start.

I very-much doubt that that would go anywhere.

Can you point me to DT features added to Linux recently that are not used by Linux?

bootph-*

No, that was applied here.

Schema wise, not sure. I took Mali GPU binding before there was any upstream driver, but that was sometime back now. Frankly, the issue rarely comes up.

Right.

[sjg20]

Also, for review, isn't it just a case of copying the linux-mtd list, or something like that?

To clarify something you mentioned earlier, are you removing the files in this repo from Linux? Does that mean that Linux pulls in its own schema as well as this one, then does the validation with the merged schema?

[ConchuOD]

Can you point me to DT features added to Linux recently that are not used by Linux?

bootph-*

No, that was applied here.

Dunno of other examples, since I don't interact with them, but the riscv,pmu stuff is not consumed by linux.

[sjg20]

Another thought I had would be to create a new partition-table type, e.g. compatible = "u-boot,binman", then put the existing schema in there, with all the subnodes, etc. Would that be a better approach? It would then be an alternative to fixed-layout, but significantly enhanced and with a larger overall goal.

I'd really like to unblock this effort somehow.

For now I can send a v4 series with the mtd people on cc.

[robherring]

Can you point me to DT features added to Linux recently that are not used by Linux?

bootph-*

No, that was applied here.

I mean the .dts files using it.

[robherring]

Also, for review, isn't it just a case of copying the linux-mtd list, or something like that?

And MTD maintainers/reviewers. It's whatever get_maintainers.pl tells you.

To clarify something you mentioned earlier, are you removing the files in this repo from Linux? Does that mean that Linux pulls in its own schema as well as this one, then does the validation with the merged schema?

It combines schemas from multiple locations, but doesn't merge contents. If the same file exists in both places, the one NOT in dtschema is used. dt-mk-schema is what does this and processes all the schemas (generates 'select' schemas, transforms scalar and array schemas to matrix (since we have no type info and everything is decoded the same way), etc.). That file, processed-schema.json, is also available from my CI job (https://gitlab.com/robherring/linux-dt/-/jobs/5108195837/artifacts/browse).

[sjg20]

Also, for review, isn't it just a case of copying the linux-mtd list, or something like that?

And MTD maintainers/reviewers. It's whatever get_maintainers.pl tells you.

OK, thanks, sent.

To clarify something you mentioned earlier, are you removing the files in this repo from Linux? Does that mean that Linux pulls in its own schema as well as this one, then does the validation with the merged schema?

It combines schemas from multiple locations, but doesn't merge contents. If the same file exists in both places, the one NOT in dtschema is used. dt-mk-schema is what does this and processes all the schemas (generates 'select' schemas, transforms scalar and array schemas to matrix (since we have no type info and everything is decoded the same way), etc.). That file, processed-schema.json, is also available from my CI job (https://gitlab.com/robherring/linux-dt/-/jobs/5108195837/artifacts/browse).

OK, thanks, I will try to remember that.

[sjg20]

@robherring I forgot to mention that I went back to the v1 patch for the Linux submission. There are various other random formats in there which have compatible strings, so I thought it easier to focus on Binman than trying to define a firmware layout for the whole industry.

[robherring]

Moved to the existing MTD bindings in the linux tree.