manup commented 1 year ago

DDF bundled format

Important PRE-ALPHA-ALPHA this is only a collection of thoughts to figure out how it could be handled.

Goals

Easy exchange of DDFs
Everything bundled in one DDF and being self contained
Having multiple versions of external scripts running (with same file path)
Fast to index
Easy to extend without breaking backward compatibility
Signatures to determine if a bundle is official, testing

Base DDFs

Currently everything works on base DDF files with file extension .jsonwhich can reference external scripts, for example: the script to parse the Tuya version exists only once and multiple Tuya DDFs do reference it. If the DDF is copied to another installation, all the scripts need to be copied as into respective locations.

While developing DDFs nothing will change here, but a new format is introduced which bundles up all dependencies into one self contained file, very similar to how Apps work on macOS, see: https://en.wikipedia.org/wiki/Bundle_(macOS).

DDF bundle

This file format has the extension .ddf and is generated from a base DDF with the deCONZ DDF Editor or via command line, it's not meant to be edited by hand.

The format is a Resource Interchange File Format (RIFF), see: https://en.wikipedia.org/wiki/Resource_Interchange_File_Format Which can store arbitrary content and more importantly can by extended easily while being backward compatible. All multi byte values are little-endian encoded.

U32 DDF_BUNDLE_MAGIC
U32 Tag
U32 Size
Data[Size]
....
U32 Tag
U32 Size
Data[Size]

DDF_BUNDLE_MAGIC is ASCII "DDFB"

Chunks

1. Descriptor

Tag: "DESC"

This is always the first chunk and allows fast indexing and matching without parsing the whole DDF. For the format we might use JSON or some simple key-value scheme?

{
  "last_modified": "2023-02-08T23:20z",
  "version": "v1.0.1",
  "min.deconz": "v2.19.3",
  "schema": "devcap1.schema.json",
  "product": "acme 2000",
  "forum": "url-to-forum-entry",
  "ghissue": "url-to-github-entry",
  "vp": [
    ["Philips", "acme 2000"],
    ["Signify", "acme 200"]
  ],
  "other": "stuff"
}

last_modified is the freshest time stamp over the base DDF as well as external Javascript.

Not sure if we want to bake all that into here, but could be nice to build a meaningful UI.

or key value based?

date: 2023-02-08T23:20z\n
version: v1.0.1
v: Philips\n
p: acme 2000\n
v: Signify\n
p: acme 2000\n
...

The most important part here is that only this chunk needs to be parsed when looking for a device.

2. DDF JSON (compressed)

Tag: "DFFC"

Holds the base DDF compressed with zlib.

3. External File

Tag: "EXTF"

U16 PathLength
U8 [PathLength] filepath (UTF8 encoded and '\0' zero terminated for easy printing)
U32 ModificationTime
U32 FileSize
U8 Data[FileSize]

For example the following DDF snipped references a external Javascript file:

"parse": {
  "at": "0x0008",
  "cl": "0x0300",
  "ep": 10,
  "fn": "zcl",
  "script": "../generic/color_control_cluster/parse_color_mode.js"
}

While loading the DDF deCONZ matches the "script" path to the respective chunk and loads the content. Internally only one copy for (file path, content, sha256(content)) is kept in memory so we don't end up with dozens of duplicates but are still able to have multiple versions running for the same path.

Note: The file content is not written to the file system and only being hold in RAM.

Some other files we could embed this way:

device image
changelog some_path_ending_with_changelog.md
pairing instructions some_path_ending_with_pairing.md
known issues some_path_ending_with_known_issues.md

Perhaps these files should be all combined into one device Markdown document?

5. Signatures

Tag: "SIGN"

Holds one or more signatures over all previous chunks. The signature and public key use the secp256k1 elliptic curve format.

u8[32] PublicKey
u8[64] Signature

Number of entries is ChunkSize / (32 + 64).

Signatures are very easy to handle with a few lines of code and make sure the DDF is not messed with. A DDF bundle which is submitted for testing can be promoted to stable / official by simply adding another signature in this chunk nothing else needs to be modified.

Signature can also be used when filtering for "official".

Unpacking

While the main goal of the bundle format is easier distribution it's also simple to restore the original base DDF and referenced files in the correct place via command line $ ddf-unpack some-device.ddf.

Other considerations

Bundle everything in one large JSON file

Pros

JSON is simple

Cons

the base DDF needs to be modified or embedded in a parent JSON object
external references need to be base64 encoded and embedded
slow to index
signatures can't be embedded

Zehir commented 1 year ago

For the External File I would like to add a File type before the path: U8[4] File type SCJS , CHLG, NOTE

SCJS = Script in javascript
CHLG = Changelog in markdown
NOTI = Note, info, ... in markdown
NOTW = Note, warnings, ... in markdown
KWIS = Know issue in markdown
UBIN = Update binary

About the patch, why (UTF8 encoded and '\0' zero terminated for easy printing) why by adding a zero character can help printing ?

For description I would keep Json because it's already used by DDF and by the rest API. Lets keep the format constant. I would like to replace vp by deviceidentifier because it's more or less what this is.

Zehir commented 1 year ago

About the DESC content I would like to change some properties:

{
  "last_modified": "2023-02-08T23:20z",
  "version": "v1.0.1",
  "min.deconz": "v2.19.3",
  "schema": "devcap1.schema.json",
  "product": "acme 2000",
  "forum": "url-to-forum-entry",
  "ghissue": "url-to-github-entry",
  "vp": [
    ["Philips", "acme 2000"],
    ["Signify", "acme 200"]
  ],
  "other": "stuff"
}

last_modified

Add the complete time definition : 2023-01-08T17:24:24z

product

Allow the source DDF product property to be an array to have multiple name to allow rebrander tuya device that use the same DDF. Or when the name depend of the language. In that case add a language prefix ?

min.deconz

Rename it to version.deconz and use the same syntax as npm : https://semver.npmjs.com/ For example: "version.deconz": "^2.19.3" to allow anything after 2.X but not 3.X

version

Current DDF version using semantic versioning instead of arbitrary string.

schema

It's the schema of the DDF or of the Bundle ? I don't think we need the DDF schema here. I don't we why we would need to filter on that.

status

Add the status of the DDF that will be confirmed by the signature

forum and ghissue

Maybe instead the store page ?

manup commented 1 year ago

last_modified

Add the complete time definition : 2023-01-08T17:24:24z

Good point, yes should be the full format.

product

Allow the source DDF product property to be an array to have multiple name to allow rebrander tuya device that use the same DDF. Or when the name depend of the language. In that case add a language prefix ?

The product key in the actual DDF is just a human readable name for the modelid, especially when there are cryptic strings in the Basic Cluster. For example for manufacturer name "Philips" and modelid "SML001" the product is "Hue motion sensor". The creator of the DDF specifies this.

In the DDF the vendor key solves the same purpose to provide a human readable string for manufacturer name.

Handling brands is difficult if not impossible, since often same brands of Tuya might use the same manufacturer name and modelid, currently I'd like to avoid making these two arrays.

Localization would be nice, and could be embedded but this might be better handled in a separate Weblate interface?

{
  "product" : "Hue motion sensor",
  "product[fr]": "Hue détecteur de mouvement"
}

(stolen from how .desktop files localize keys)

Not sure ...

min.deconz

Rename it to version.deconz and use the same syntax as npm : https://semver.npmjs.com/ For example: "version.deconz": "^2.19.3" to allow anything after 2.X but not 3.X

version.deconz is fine, but note that the next major version change won't break any bundles, internally here the version is treated as minimal version, backwards compatibility is a must, the 3.x is all about UI changes.

version

Current DDF version using semantic versioning instead of arbitrary string.

schema

It's the schema of the DDF or of the Bundle ? I don't think we need the DDF schema here. I don't we why we would need to filter on that.

It's the DDF schema, we can ditch it from here, it would only be used internally in deCONZ.

status

Add the status of the DDF that will be confirmed by the signature

Yes status can be copied here from the DDF, it might be incorrect or misleading since everybody can create a DDF/bundle and mark it stable, only the signatures tell the whole story :)

forum and ghissue

Maybe instead the store page ?

Yeah was just an idea, initially we should keep it simple, store page is fine.

Zehir commented 1 year ago

product

Allow the source DDF product property to be an array to have multiple name to allow rebrander tuya device that use the same DDF. Or when the name depend of the language. In that case add a language prefix ?

The product key in the actual DDF is just a human readable name for the modelid, especially when there are cryptic strings in the Basic Cluster. For example for manufacturer name "Philips" and modelid "SML001" the product is "Hue motion sensor". The creator of the DDF specifies this.

In the DDF the vendor key solves the same purpose to provide a human readable string for manufacturer name.

Handling brands is difficult if not impossible, since often same brands of Tuya might use the same manufacturer name and modelid, currently I'd like to avoid making these two arrays.

Localization would be nice, and could be embedded but this might be better handled in a separate Weblate interface?
{
  "product" : "Hue motion sensor",
  "product[fr]": "Hue détecteur de mouvement"
}
(stolen from how .desktop files localize keys)

Not sure ...

Yes why not it's could work, but I need all in the desc to be able to filter and display infos. But it's could be extracted from the DDF when it's uploaded and not part of the description. Or keep the manufacturername and modelid in the DDF and the vendor and product in the desc ?

  "manufacturername": "SMaBiT",
  "modelid": "AV2010/21C",
  "vendor": "SMaBiT",
  "product": "AV2010/21C Flat Magnetic Contact",

min.deconz

Rename it to version.deconz and use the same syntax as npm : https://semver.npmjs.com/ For example: "version.deconz": "^2.19.3" to allow anything after 2.X but not 3.X

version.deconz is fine, but note that the next major version change won't break any bundles, internally here the version is treated as minimal version, backwards compatibility is a must, the 3.x is all about UI changes.

Then it's >=2.19.3 but I would like to support all format in case we need to limit a maximum version at some point too

version

Current DDF version using semantic versioning instead of arbitrary string.

schema

It's the schema of the DDF or of the Bundle ? I don't think we need the DDF schema here. I don't we why we would need to filter on that.

It's the DDF schema, we can ditch it from here, it would only be used internally in deCONZ.

ok

Zehir commented 1 year ago

Why the DDF chunk is named DFFC and not DDFC ?

Zehir commented 1 year ago

I moved the file definition here : https://github.com/deconz-community/ddf-tools/blob/main/packages/bundler/README.md It's will be easier to read and If we need to change the definition edit the file there. It's was starting to be complicated to read each response every time.

Main changes are:

Added the file type on the EXTF chunk Replace the ModificationTime format from timestamp to ISO 8601, I know it's use more space but it's to avoid the Year 2038 problem. Renamed the tag of the DDF file from DFFC to DDFC In the DESC chunk rename vp to device_identifiers and add the links property. Fix some cases to use Snake case.

manup commented 1 year ago

Thanks a lot, it's a good write up for the format :+1: Perhaps we should leave the DESC chunk uncompressed as it's already small in most cases. For the file types I was thinking about using mime types but the tags work as well, and are smaller :)

To be honest I'd not include the OTA files, they are kind of RIFF files on their own with an average size of 250 Kb and better kept separate at least for now.

Side note: The button maps will very likely change as our current approach with the static JSON representation is super limited and half way kept in a monster button_maps.json file. I have a prototype which uses a very small and fast bytecode VM instead, but not everyone likes to write assembly. Currently in DDFs we workaround via Javascript for some switches like Philips Hue which need more than the static JSON but it's not the best solution yet.

ebaauw commented 1 year ago

Can we please get rid of the . in version.deconz? Either use a sub-object "version": { "deconz": ... } or replace by a _, version_deconz.

Also, for localisation, I would prefer a separate sub-object, like:

  "productname": "non-localised product name",
  "productname_localised": {
    "de": "german product name",
    "en": "english product name",
     "fr": "french product name",
  }

Zehir commented 1 year ago

Thanks a lot, it's a good write up for the format 👍 Perhaps we should leave the DESC chunk uncompressed as it's already small in most cases. For the file types I was thinking about using mime types but the tags work as well, and are smaller :) Yep and 4 characters is enough for the future

To be honest I'd not include the OTA files, they are kind of RIFF files on their own with an average size of 250 Kb and better kept separate at least for now.

Sure, we could instead store the OTA file url in the DESC data when we would do it.

Side note: The button maps will very likely change as our current approach with the static JSON representation is super limited and half way kept in a monster button_maps.json file. I have a prototype which uses a very small and fast bytecode VM instead, but not everyone likes to write assembly. Currently in DDFs we workaround via Javascript for some switches like Philips Hue which need more than the static JSON but it's not the best solution yet.

Yes we need something like DDF but for buttons maps and merge back the data from buttons map to the bundle.

Zehir commented 1 year ago

Can we please get rid of the . in version.deconz? Either use a sub-object "version": { "deconz": ... } or replace by a _, version_deconz.

Yes I already change that 👍

Also, for localisation, I would prefer a separate sub-object, like:
  "productname": "non-localised product name",
  "productname_localised": {
    "de": "german product name",
    "en": "english product name",
     "fr": "french product name",
  }

works for me

Zehir commented 1 year ago

I have updated the document : https://github.com/deconz-community/ddf-tools/blob/main/packages/bundler/README.md

Zehir commented 1 year ago

I would like to add the size of the bundle after the DDFB and also moving the SIGN outside of the DDFB chunk to be able to quickly find the signature position in the bundle. And also the signature and external file chunk size.

Thoses sizes are not required but if we want to stick to the RIFF format we probably should add them.

So the signature are only the dark blue part.

manup commented 1 year ago

I agree that's a good approach to quickly seek around the data. Also that signatures are for the DDFB content makes a lot of sense, so we could add more signatures without changing the content.

Zehir commented 1 year ago

Since all chunks are uniques exept EXTF should we put the EXTFs chunks in a LIST chunk ?

Zehir commented 1 year ago

I updated the signature format if you can take a look at : https://github.com/deconz-community/ddf-bundler/commit/eb9665f129c80b8527258838ec8a1682cf137322#diff-9d3e3048bcd6cfd861d661f988addc8ce0b9ef6e0e9c41073f8748232cf712ef

Zehir commented 1 year ago

I made more change about the signature :

See : https://github.com/deconz-community/ddf-tools/blob/main/packages/bundler/README.md

About the things lengths that are small like signature length. I keep them in a U16 format even if a U8 is enought just to be easier for coding because all chunk size are on U32 and data size are on U16.

Zehir commented 1 year ago

I updated the playground with the lastest changes ; https://deconz-community.github.io/ddf-tools/

dresden-elektronik / deconz-rest-plugin-v2

DDF bundle format (RFC) #15

DDF bundled format

Goals

DDF bundle

Chunks

1. Descriptor

2. DDF JSON (compressed)

3. External File

5. Signatures

Unpacking

Other considerations

Bundle everything in one large JSON file

last_modified

product

min.deconz

version

schema

status

forum and ghissue

last_modified

product

min.deconz

version

schema

status

forum and ghissue

product

min.deconz

version

schema