add support for custom object in tuf targets metadata

dennisvang commented 8 months ago

Description

TUF targets metadata objects support an optional custom object, which can be used to specify additional information about the individual target files.

From the TUF spec:

CUSTOM

An object. If defined, the elements and values of the CUSTOM object will be made available to the client application. The format of the CUSTOM object is opaque to the framework, which only needs to know that the "custom" attribute maps to an object. The CUSTOM object may include version numbers, dependencies, requirements, or any other data that the application wants to include to describe the file at TARGETPATH. The application may use this information to guide download decisions.

This offers endless possibilities, for example:

include information about the update that can be displayed to the user, e.g. a summary of changes
include information for verifying the integrity of reconstructed archives after patch application
specify additional actions to be taken during an update
...

Implementation

This requires changes both on the repo side and on the client side:

Repo: Ability to specify custom metadata when adding a new target. This implies we need a custom metadata arg in Repository.add_bundle().
Client: Ability to retrieve the custom metadata when checking for updates. The Client.check_for_updates() method returns a TargetMeta, which looks like the perfect place to include the custom metadata. DO NOTE: the TargetMeta object returned is always that of the full archive, even if a patch update will be applied. So, on the repo-side, any custom metadata required by the user application would need to be added to the archive target, not the patch target.

Notes

In python-tuf (tuf.api.metadata), the custom object is represented by the TargetFile.custom property.

Judging from the source, the custom property can be set by adding a 'custom' member to the TargetFile.unrecognized_fields dictionary. For example:

target = TargetFile.from_file(...)
target.unrecognized_fields['custom'] = dict(...)

EDIT:

Actually, all unrecognized_fields are included in the metadata, as can be seen in the source. So, custom is nothing special in that respect, but it's the only "unrecognized field" that's defined in the spec and has a corresponding property.

Also note that python-tuf does not check whether the value assigned to custom is actually a dict (which maps to a JSON object). The spec does not provide an explicit definition of the term object, but it uses "a subset of the JSON object format", so we'll assume object implies the equivalent of a JSON object (which is a dict in Python).

dennisvang commented 7 months ago

Although #100 works, it would probably be more convenient only to allow custom metadata for archives.

This would simplify the interface, and it makes more sense, because all the client app will ever see is the TargetMeta for the archive (not for the patch).

Perhaps keep the patch option for internal use?

dennisvang commented 6 months ago

In some cases it is convenient to store custom metadata for tufup's internal use.

Examples are the tar size and hash data from #105 and the "required" flag from #111.

To prevent key clashes, it would be preferable to separate this internal custom metadata from any user-specified custom metadata.

Moreover, the internal metadata should be hidden from casual users, to prevent confusion.

The cleanest approach would probably be to separate the TUF CUSTOM object into a TUFUP object and a USER object, as follows:

{
  "custom": {
    "tufup": {},
    "user": {}
  }
}

Methods need to be updated to ensure the casual user is only aware of the USER object.

dennisvang / tufup