theupdateframework / python-tuf

Python reference implementation of The Update Framework (TUF)
https://theupdateframework.com/
Apache License 2.0
1.63k stars 272 forks source link

Trust Pinning #361

Closed ecordell closed 7 years ago

ecordell commented 8 years ago

Proposal

Proposal #: 2 Title: Trust Pinning Author: Evan Cordell, Jake Moshenko, Justin Cappos, Vladimir Diaz, Sebastien Awwad Created: August 18, 2016 Type: Design Status: Draft

Rationale

This is a proposal of a design for pinning trusted keys, which would allow clients to root their trust at a namespace (or lower) delegated set of keys instead of the repository root.

An example use case: I trust django's published public keys to have signed off on django, but I don't necessarily trust PyPI to remain uncompromised (and possibly convince me of different public keys for django).

This proposal also addresses the problem of private metadata. In the current TUF spec, there is no way to hide all information about the existence of other metadata in the system. This is a problem in a multi-tenant scenario where knowledge of metametadata could be sensitive (e.g. timing of creating a target, names of targets, etc).

Overview

We introduce a new file, pinned.json, which permits users to pin root files and associate them with a particular namespace prefix. If an entry matches the current package, the pinned root must be used. If it does not match, there is a fallback to the global root.

This constructs two (or more) trust paths for target files, and allows the user to pick between them. Clients that trust the global root (e.g. PyPI) will trust all packages served by it, and those that wish to root trust with a namespace owner (e.g. django project) can pin to those keys. See this diagram for an example.

Because the pinning mechanism uses roots, the "pinned" keys may be rotated according to the standard root rotation scheme. In that sense, you are pinning the root of a tree of keys which can grow over time, rather than pinning a set of keys that must never change.

Pin File

pinned.json maps a prefix to a location where the pinned root can be found and an optional url for updating it. This file is not available from a TUF repository. It is either constructed by explicit actions from the client (e.g. "pin this role's keys") or by an out-of-band bootstrap (e.g. "here's our organization's pinned.json").

 {
   "django": {
      "location": "pinned/django",  // default pinned directory structure
      "url": "https://www.djangoproject.com/release/metadata"
   },
   "private-requests-beta": {
      "location": "pinned/requests"
       // no url - can't be updated automatically
   },
   "private-flask-beta": {
      "location": "/usr/local/evan/flask-beta" // metadata can live elsewhere if desired
   }
 }

Pinned Metadata

Pinned metadata lives in a specific default directory, sharing the same layout as a "normal" repo but nested within a prefix namespace, e.g.

pinned
└── django  // prefix
    ├── root.json
    ├── snapshot.json
    ├── targets.json
    └── timestamp.json

This can be changed with the location field of the pinned.json file, which may be useful if e.g. sharing a network drive.

Complex ACLs can be enforced and/or bootstrapped by sending a user an appropriately generated pinned.json, noting that any metadata endpoint (root repo, or any pinned repo) can have its own access control mechanism.

Hiding

A private package can be omitted from the primary hierarchy entirely, having its own snapshot and target files separate from those provided with root. The snapshot.json and target.json could be signed with the same snapshot and target keys used for the public parts of the repository, or they can be managed and signed by the owner of the private delegated role. Access to these private roles is granted by sending the metadata to the appropriate users (further restricted by ACLs if needed). A url pointing to where the snapshot and timestamp can be found is added to the pinned.json file in the case of private roles.

Hard Pinning

Hard pinning, in which a specific set of non-changing keys are used, can be accomplished by creating the a pinned metadata repository and not specifying a url. Without a url, nothing can convince a client to use different keys. This may be useful for priming a box for a one-time initial pull (with the assumption that it will be killed rather than updated directly).

Repository structure

With this pinning structure it makes sense to structure namespaces and/or packages with their own roots. Alternately, a user can generate a root for a given package/target delegation locally if it doesn't exist, by generating keys locally and signing.

Because a delegation is also a target file, a global root can delegate to target files of other repos. This allows a simple way to provide both global and namespaced target files.

Prior Art

It's worth mentioning that Notary has a pinning implementation currently. Although this proposal differs and has slightly different goals, the Notary format should be compatible with this through a simple transformation.

trishankkarthik commented 8 years ago

There may (or may not) be some useful ideas from this paper on how to rotate keys.

Baton: Key Agility for Android without a Centralized Certificate Infrastructure

JustinCappos commented 8 years ago

I think you left off the bit about how and where the snapshot and timestamp information is placed in the hiding paragraph. However, from the example, it looks like you mean to add it (optionally) in the pinned.json file. If it isn't there, does this information live in tuf.conf?

Where should information about the snapshot and timestamp for a pinned role live and how is this information updated?

Also, where is the information about the roles that sign django and requests in your example? Is this stored in the pinned.json?

When / how can pinned.json be changed? For example, what if the location of the timestamp / release needs to change?

Not that I am promoting this design instead, but I'd like to see a comparison with just having a pinned file that associates different target patterns with different root files. The idea is that the rest of TUF would proceed normally after selecting which root file to use.

Note: while I am generally supportive of adding pinning to TUF, I do think we need to really carefully think through all of the boundary cases...

On Thu, Aug 18, 2016 at 1:49 PM, Evan Cordell notifications@github.com wrote:

This is a proposal of a design for pinning trusted keys, which would allow clients to root their trust at a namespace (or lower) delegated set of keys instead of the repository root.

An example use case: I trust django's published public keys to have signed off on django, but I don't necessarily trust PyPI to remain uncompromised (and possible convince me of different public keys for django).

This proposal also addresses the problem of private metadata. In the current TUF spec, there is no way to hide all information about the existence of other metadata in the system. This is a problem in a multi-tenant scenario where knowledge of metametadata could be sensitive (e.g. timing of creating a target, names of targets, etc). Proposal Overview

Allow users to associate "pinned" keys with a particular "namespace" (target delegation). Also have a fallback pointing to the repository root's keys. This means that clients who trust the registy get a seamless experience, and clients who need to root trust at other levels can do so easily. Lemma 1

Any time a delegated role (targets/django) needs to be updated (either because of changes to target files or because of key rotations), a new set of metadata (targets/django') is created, and the original (targets/django) delegates to the new (targets/django'). This occurs for every update, with old versions continually delegating to new versions e.g A -> B -> C -> D where -> implies delegation. This is an important prerequisite for the following reason: if a client trusts the keys in A, then it will trust the keys in B. This means that a client can "pin" trust to any set of keys in the chain, while still allowing key owners to rotate when necessary. Lemma 2

Introduce a new metadata file, pinned.json which simply maps role names to target roles.

{ "django": { "target": "targets/django1.7.json", }, "requests": { "target": "targets/requests1.3.json, }, "private-requests-beta": { "target": "targets/requests1.4beta.json, "url": "https://url/of/private/snapshot/and/timestamp", } }

Proposal

By combining the behavior from lemma 1 and the new structure defined in lemma 2 we achieve two goals: pinning and information hiding.

pinning: The role to target file mappings in pinned.json are considered pinned keys. If the user wishes to pin a role's keys, they simply add an entry to pinned.json. This will pin the keys for particular version of the metadata. When the metadata is updated according to lemma 1, the keys referenced will be updated accordingly. We enforce that if a role exists in the pinned key file, new keys will not be trusted just because other metadata (root, snapshot, etc) updates the keys identified for that role.

hiding: A private delegated role can be omitted from the primary hierarchy entirely, having its own snapshot and target files separate from those provided with root. The snapshot.json and target.json could be signed with the same snapshot and target keys used for the public parts of the repository, or they can be managed and signed by the owner of the private delegated role. Access to these private roles is granted by sending the metadata to the appropriate users (further restricted by ACLs if needed). A url pointing to where the snapshot and timestamp can be found

What do we think?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD4xd4X31Ywjca66GjZgI5dNb2vnoks5qhJsigaJpZM4Jnvng .

ecordell commented 8 years ago

I think you left off the bit about how and where the snapshot and timestamp information is placed in the hiding paragraph. However, from the example, it looks like you mean to add it (optionally) in the pinned.json file. If it isn't there, does this information live in tuf.conf?

Yes, I was thinking you would enter a url where it could be found in pinned.json. Perhaps it should be either a url or a local file path?

Where should information about the snapshot and timestamp for a pinned role live and how is this information updated?

I think it makes sense to keep it in pinned.json. I guess there should be an additional pinned folder with a directory structure to support storing/updating snapshot and timestamp.

When / how can pinned.json be changed? For example, what if the location of the timestamp / release needs to change?

IMO, this should be an explicit user change with a local tool. Pinning is an explicitly out-of-band operation to begin with.

Not that I am promoting this design instead, but I'd like to see a comparison with just having a pinned file that associates different target patterns with different root files. The idea is that the rest of TUF would proceed normally after selecting which root file to use.

Here's a diagram of the proposed pinning scheme. In this case, the delegation key django.key only needs to sign one piece of metadata, just like delegations currently work in TUF. (dashed arrows indicate a key signing metadata)

Here's a diagram of pinning roots instead. This also works, but it requires the holder of the delegation key to sign both a pinned root file and a target delegation file. It becomes the onus of the delegation key holder to "allow" pinning by signing both sets of data, whereas the other case can be entirely a client decision.

In the case where the delegation key holder is also signing the snapshots (rather than handing that responsibility to e.g. PyPI) then both require some action by the delegation key holder.

There's actually something that could be quite nice if a certain delegation pattern were enforced: a picture may help. Basically, if you enforce that projects must have delegation keys, then you can construct two "paths" to them; one public and rooted at the root.json, one "private" (either actually private or just published in a different location). You can have actually private metadata if the public path is removed.

JustinCappos commented 8 years ago

Those diagrams are really helpful. Thanks for making them!

So, in the "proposed pinning scheme", wouldn't the snapshot and timestamp file for django be listed in the pinned file?

Also, I think that having roots pinned is equivalent to pinning targets in terms of whether the role that is pinned needs to be involved. If I want to pin your target file (without you knowing it), I can just create my own root file that lists your targets key and use it, right? Doesn't this behave identically to the proposed targets pinning scheme?

To me, it really seems like the difference is whether the pinning file lists the timestamp / release or whether there are root files that do so. Root files seem to make it easier for the pinned role to update their snapshot / timestamp / targets keys. Is there another major difference I'm missing?

On Mon, Aug 22, 2016 at 3:41 PM, Evan Cordell notifications@github.com wrote:

I think you left off the bit about how and where the snapshot and timestamp information is placed in the hiding paragraph. However, from the example, it looks like you mean to add it (optionally) in the pinned.json file. If it isn't there, does this information live in tuf.conf?

Yes, I was thinking you would enter a url where it could be found in pinned.json. Perhaps it should be either a url or a local file path?

Where should information about the snapshot and timestamp for a pinned role live and how is this information updated?

I think it makes sense to keep it in pinned.json. I guess there should be an additional pinned folder with a directory structure to support storing/updating snapshot and timestamp.

When / how can pinned.json be changed? For example, what if the location of the timestamp / release needs to change?

IMO, this should be an explicit user change with a local tool. Pinning is an explicitly out-of-band operation to begin with.

Not that I am promoting this design instead, but I'd like to see a comparison with just having a pinned file that associates different target patterns with different root files. The idea is that the rest of TUF would proceed normally after selecting which root file to use.

Here's a diagram of the proposed pinning scheme http://knsv.github.io/mermaid/live_editor/#/view/JSUgUGlubmluZyBkZWxlZ2F0aW9ucwpncmFwaCBMUgogICAgUm9vdC5rZXkgLS4tPiBBCiAgICBBW1BZUEkgcm9vdC5qc29uXSAtLT4gdGltZXN0YW1wLmpzb24KICAgIEEgLS0-IHNuYXBzaG90Lmpzb24KICAgIEEgLS0-IHRhcmdldHMuanNvbiAKICAgIHRhcmdldHMuanNvbiAtLT4gQlt0YXJnZXRzL2RqYW5nby5qc29uIGRlbGVnYXRpb25dCiAgICBwaW5uZWQuanNvbiAtLT4gRltkamFuZ29dCiAgICBGIC0tPiBDW2RqYW5nby90aW1lc3RhbXAuanNvbl0KICAgIEYgLS0-IERbZGphbmdvL3NuYXBzaG90Lmpzb25dCiAgICBEamFuZ28ua2V5IC0uLT4gQgogICAgRiAtLT4gQgogICAgcGlubmVkLmpzb24gLS0-ICoKICAgICogLS0-IEE. In this case, the delegation key django.key only needs to sign one piece of metadata, just like delegations currently work in TUF. (dashed arrows indicate a key signing metadata)

Here's a diagram of pinning roots instead http://knsv.github.io/mermaid/live_editor/#/view/JSUgUGlubmluZyByb290cwpncmFwaCBMUgogICAgUm9vdC5rZXkgLS4tPiBBCiAgICBBW1BZUEkgcm9vdC5qc29uXSAtLT4gdGltZXN0YW1wLmpzb24KICAgIEEgLS0-IHNuYXBzaG90Lmpzb24KICAgIEEgLS0-IHRhcmdldHMuanNvbiAKICAgIHRhcmdldHMuanNvbiAtLT4gQlt0YXJnZXRzL2RqYW5nby5qc29uIGRlbGVnYXRpb25dCgogICAKICAgIEZbRGphbmdvIHJvb3QuanNvbl0gLS0-IEdbZGphbmdvL3RpbWVzdGFtcC5qc29uXQogICAgRiAtLT4gSFtkamFuZ28vc25hcHNob3QuanNvbl0KICAgIEYgLS0-IElbZGphbmdvL3RhcmdldHMuanNvbl0KCiAgICBwaW5uZWQuanNvbiAtLT4gRgogICAgcGlubmVkLmpzb24gLS0-IEEKICAgIERqYW5nby5rZXkgLS4tPiBCCiAgICBEamFuZ28ua2V5IC0uLT4gRg. This also works, but it requires the holder of the delegation key to sign both a pinned root file and a target delegation file. It becomes the onus of the delegation key holder to "allow" pinning by signing both sets of data, whereas the other case can be entirely a client decision.

In the case where the delegation key holder is also signing the snapshots (rather than handing that responsibility to e.g. PyPI) then both require some action by the delegation key holder.

There's actually something that could be quite nice if a certain delegation pattern were enforced: a picture may help http://knsv.github.io/mermaid/live_editor/#/view/JSUgUGlubmluZyByb290cwpncmFwaCBMUgogICAgUm9vdC5rZXkgLS4tPiBBCiAgICBBW1BZUEkgcm9vdC5qc29uXSAtLT4gdGltZXN0YW1wLmpzb24KICAgIEEgLS0-IHNuYXBzaG90Lmpzb24KICAgIEEgLS0-IHRhcmdldHMuanNvbiAKICAgIHRhcmdldHMuanNvbiAtLT4gQlt0YXJnZXRzL2RqYW5nby5qc29uIGRlbGVnYXRpb25dCgogICAKICAgIEZbRGphbmdvIHJvb3QuanNvbl0gLS0-IEdbdGltZXN0YW1wLmpzb25dCiAgICBGIC0tPiBIW3NuYXBzaG90Lmpzb25dCiAgICBGIC0tPiBJW3RhcmdldHMuanNvbl0KICAgIEkgLS0-IEIKCiAgICBwaW5uZWQuanNvbiAtLT4gRgogICAgcGlubmVkLmpzb24gLS0-IEEKICAgIERqYW5nb09yZ1Jvb3Qua2V5IC0uLT4gRgogICAgRGphbmdvUHJvamVjdC5rZXkgLS4tPiBC. Basically, if you enforce that projects must have delegation keys, then you can construct two "paths" to them; one public and rooted at the root.json, one "private" (either actually private or just published in a different location). You can have actually private metadata if the public path is removed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-241526091, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD2AvRkUcvNuYy4G-VsqLyN72M84Dks5qiftPgaJpZM4Jnvng .

ecordell commented 8 years ago

So, in the "proposed pinning scheme", wouldn't the snapshot and timestamp file for django be listed in the pinned file?

My thought was that this would be a convention-based location on the filesystem, e.g. if you've pinned projects prefixed with django then there's a folder structure like:

pinned
└── django
    ├── snapshot.json
    ├── targets.json
    └── timestamp.json

I was considering the exact structure an implementation detail but in retrospect it would help discussion to pin it down (pun intended).

The url field was also suggested as a way to indicate where to fetch new metadata for pinned roles if desired. The fetching scheme would need to retain the forward-signing properties of root files for proper key rotation, which does suggest that working with roots might be the right approach here.

If I want to pin your target file (without you knowing it), I can just create my own root file that lists your targets key and use it, right? Doesn't this behave identically to the proposed targets pinning scheme?

I believe so, yes. I hadn't considered that as an option but after consideration I believe that retains the properties we want. Neat!

Root files seem to make it easier for the pinned role to update their snapshot / timestamp / targets keys.

I agree with this. Pinned targets need the same forward signing anyway so making them roots makes sense.

All of this discussion in mind, I think some revision to the original proposal is in order:

 {
   "django": {
      "location": "pinned/django",  // default pinned directory structure
      "url": "https://www.djangoproject.com/release/metadata"
   },
   "private-requests-beta": {
      "location": "pinned/requests"
       // no url - can't be updated automatically
   },
   "private-flask-beta": {
      "location": "/usr/local/evan/flask-beta" // metadata can live elsewhere if desired
   }
 }
pinned
└── django  // prefix
    ├── root.json
    ├── snapshot.json
    ├── targets.json
    └── timestamp.json

This can be changed with the location field of the pinned.json file, which may be useful if e.g. sharing a network drive.

If this sounds good, I'll update the original proposal to match.

JustinCappos commented 8 years ago

Okay, great! This sounds good to me. Please feel free to start updating the proposal.

On Wed, Aug 24, 2016 at 11:13 AM, Evan Cordell notifications@github.com wrote:

So, in the "proposed pinning scheme", wouldn't the snapshot and timestamp file for django be listed in the pinned file?

My thought was that this would be a convention-based location on the filesystem, e.g. if you've pinned projects prefixed with django then there's a folder structure like:

pinned └── django ├── snapshot.json ├── targets.json └── timestamp.json

I was considering the exact structure an implementation detail but in retrospect it would help discussion to pin it down (pun intended).

The url field was also suggested as a way to indicate where to fetch new metadata for pinned roles if desired. The fetching scheme would need to retain the forward-signing properties of root files for proper key rotation, which does suggest that working with roots might be the right approach here.

If I want to pin your target file (without you knowing it), I can just create my own root file that lists your targets key and use it, right? Doesn't this behave identically to the proposed targets pinning scheme?

I believe so, yes. I hadn't considered that as an option but after consideration I believe that retains the properties we want. Neat!

Root files seem to make it easier for the pinned role to update their snapshot / timestamp / targets keys.

I agree with this. Pinned targets need the same forward signing anyway so making them roots makes sense.

All of this discussion in mind, I think some revision to the original proposal is in order:

  • pinned.json maps a prefix to a location where the pinned root can be found and an optional url for updating it:

    { "django": { "location": "pinned/django", // default pinned directory structure "url": "https://www.djangoproject.com/release/metadata" }, "private-requests-beta": { "location": "pinned/requests" // no url - can't be updated automatically }, "private-flask-beta": { "location": "/usr/local/evan/flask-beta" // metadata can live elsewhere if desired } }

    • Pinned metadata lives in a specific default directory, sharing the same layout as a "normal" repo but nested within a prefix namespace, e.g.

pinned └── django // prefix ├── root.json ├── snapshot.json ├── targets.json └── timestamp.json

This can be changed with the location field of the pinned.json file, which may be useful if e.g. sharing a network drive.

-

We no longer need the target key rotation scheme, root rotation will apply as normal to pinned roots.

To make it so that "public" repositories do not need to be pinned to be trusted, we can set up a "root" public repo that delegates to the root keys of projects. This exploits the property that there is no difference between a target delegation and a target file. Again, a diagram illustrates this best http://knsv.github.io/mermaid/live_editor/#/view/JSUgUGlubmluZyByb290cwpncmFwaCBMUgogICAgUm9vdC5rZXkgLS4tPiBBCiAgICBBW1BZUEkgcm9vdC5qc29uXSAtLT4gdGltZXN0YW1wLmpzb24KICAgIEEgLS0-IHNuYXBzaG90Lmpzb24KICAgIEEgLS0-IHRhcmdldHMuanNvbiAKICAgIHRhcmdldHMuanNvbiAtLT4gQlt0YXJnZXRzL2RqYW5nby5qc29uXQoKICAgIEIgLS0tfEVxdWFsfCBJCgoKICAgIEZbRGphbmdvIHJvb3QuanNvbl0gLS0-IEdbdGltZXN0YW1wLmpzb25dCiAgICBGIC0tPiBIW3NuYXBzaG90Lmpzb25dCiAgICBGIC0tPiBJW3RhcmdldHMuanNvbl0KCiAgICBwaW5uZWQuanNvbiAtLT4gRgogICAgcGlubmVkLmpzb24gLS0-IEEKICAgIERqYW5nby5rZXkgLS4tPiBG. This means that clients who trust the provider (e.g. PyPI) will trust public packages by default, clients who wish to pin a project's keys may do so, and private packages can be distributed either by sending root.json files out of band or by sending a url to a separate tuf repo.

If a project matches a pinned prefix, then pinned metadata must be used to verify it. With the above scheme, there should always be a global fallback, and pinning explicitly opts out of using that fallback. This has implications for what level to pin: pinning should take place as "low" as possible, e.g. django/private/beta instead of django (so that django/public-release-1.9 doesn't require pinned data).

Complex ACLs can be enforced and/or bootstrapped by sending a user an appropriately generated pinned.json, noting that any metadata endpoint (root repo, or any pinned repo) can have its own access control mechanism.

If this sounds good, I'll update the original proposal to match.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-242100258, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XDznrQzFS7Tfh18ZF4Dy2a4o7kT1Rks5qjF-qgaJpZM4Jnvng .

ecordell commented 8 years ago

@JustinCappos Updated!

vladimir-v-diaz commented 8 years ago

I have a few comments.

  1. The overview section starts with: "We introduce a new file, pinned.json, which permits users to pin root files and associate them with a particular namespace prefix." It is not clear from this sentence alone, but is pinned.json required? What happens if the updater client cannot locate it?
  2. In the reference implementation, the TUF library is normally given a specific target to fetch securely (e.g., /packages/32/0b/Django-1.10rc1.tar.gz). Is the namespace prefix extracted from this filename, or does the software update system instead give the TUF library a project name / namespace?
  3. The TUF specification allows the trusted paths specified in metadata to include wildcards. Is this a feature that adopters / clients would want to include in pinned.json? I can see a scenario where a client would only want to use pinned keys for specific releases.
  4. What should happen if the top-level metadata (root, snapshot, and timestamp) provided by a pinned project expired? For example, if no URL is specified in pinned.json for this project and only expired metadata can be loaded locally.
  5. Should pinned.json include things like cutting delegations, multi-role delegations, etc.? What happens if a target file requested by a client is not found in the set of pinned metadata downloaded? Should the client backtrack to the global root?
  6. Do you expect the code to change much? I'd expect only the step(s) prior to downloading root.json to change, but practically nothing that follows. If the code does change in other parts of the codebase, we should discuss it.
ecordell commented 8 years ago

The overview section starts with: "We introduce a new file, pinned.json, which permits users to pin root files and associate them with a particular namespace prefix." It is not clear from this sentence alone, but is pinned.json required? What happens if the updater client cannot locate it?

I think we say that no pinned.json is the same as an empty pinned.json, i.e. all requests fall back to the configured repository root. If the updater client can't locate it, then there is effectively no pinning. I'm in favor of making this an explicit error (e.g. you specified a pinned.json file but it wasn't found, so exit).

In the reference implementation, the TUF library is normally given a specific target to fetch securely (e.g., /packages/32/0b/Django-1.10rc1.tar.gz). Is the namespace prefix extracted from this filename, or does the software update system instead give the TUF library a project name / namespace? The TUF specification allows the trusted paths specified in metadata to include wildcards. Is this a feature that adopters / clients would want to include in pinned.json? I can see a scenario where a client would only want to use pinned keys for specific releases.

I was thinking the namespace prefix would be extracted from the filename, but I can't think of a non-performance reason that it wouldn't be good to support wildcard paths. I suppose we can s/prefix/path in the proposal and allow wildcards?

What should happen if the top-level metadata (root, snapshot, and timestamp) provided by a pinned project expired? For example, if no URL is specified in pinned.json for this project and only expired metadata can be loaded locally.

This should be an invalid metadata error with appropriate messaging to the user. Pinning without a URL is a strong assertion by the user that they don't wan't different keys than they put there. No targets should be downloaded.

Should pinned.json include things like cutting delegations, multi-role delegations, etc.? What happens if a target file requested by a client is not found in the set of pinned metadata downloaded? Should the client backtrack to the global root?

I outlined this in the original proposal but I think I erased it with edits. If a target file matches a pinned path, it must be found in that pinned tree to be valid. If it's not, that's an unrecoverable error and no targets should be downloaded. Pinning should be an explicit action that signals the intent "don't fall back". A user can resolve this by updating/correcting their pinning information or removing the pin.

Do you expect the code to change much? I'd expect only the step(s) prior to downloading root.json to change, but practically nothing that follows. If the code does change in other parts of the codebase, we should discuss it.

I don't think there should be much of a change - the only thing that needs to change is the updater to understand pinned.json and pick the appropriate root. We may be want to provide additional tooling to help with writing the pinned.json file but this is not strictly required.

JustinCappos commented 8 years ago

I'll respond below. Evan, etc. please chime in if you think I'm off base or want to discuss things further.

1.

The overview section starts with: "We introduce a new file, pinned.json, which permits users to pin root files and associate them with a particular namespace prefix." It is not clear from this sentence alone, but is pinned.json required? What happens if the updater client cannot locate it?

I think we should make it required. (Originally I thought about making it optional, but the implicit delegation of '*' to some path is confusing.)

1.

In the reference implementation, the TUF library is normally given a specific target to fetch securely (e.g., /packages/32/0b/Django-1. 10rc1.tar.gz). Is the namespace prefix extracted from this filename, or does the software update system instead give the TUF library a project name / namespace?

I'm not sure I follow. Can you explain more?

1.

The TUF specification allows the trusted paths specified in metadata to include wildcards. Is this a feature that adopters / clients would want to include in pinned.json? I can see a scenario where a client would only want to use pinned keys for specific releases.

I'm in favor of allowing wildcards. For example, one may want to delegate .rpms to one role and .deb files to another. Is there a good reason to prevent this?

1.

What should happen if the top-level metadata (root, snapshot, and timestamp) provided by a pinned project expired? For example, if no URL is specified in pinned.json for this project and only expired metadata can be loaded locally.

If there is no URL for update, I think you should use the old roles, etc. It may be worth a warning, but it's not error worthy...

1.

Should pinned.json include things like cutting delegations, multi-role delegations, etc.? What happens if a target file requested by a client is not found in the set of pinned metadata downloaded? Should the client backtrack to the global root?

I think we need some of this with the way we want to use it in Uptane. We do need a pinned, multi-role delegation there. I think that Evan's use case likely would want a cutting delegation. So, we likely should just reuse this functionality from targets.

How does this complicate the design? We're not using keys and roles in the same way for the targets here...

1.

Do you expect the code to change much? I'd expect only the step(s) prior to downloading root.json to change, but practically nothing that follows. If the code does change in other parts of the codebase, we should discuss it.

I do not think anything should change apart from what we said above. If so, I'd like to discuss further.

Thanks, Justin

ecordell commented 8 years ago

I don't follow the bits about including delegations in pinned.json. Wouldn't those simply be in the targets.json? Are you considering the case where there are delegation roles shared across multiple repos?

JustinCappos commented 8 years ago

Yes, I'm thinking of the case where one wants to say that a target must appear under two different roots. But you want both roots to need to be compromised for the users to be harmed.

This did come up for us in another context, so we'd like to support that case.

On Mon, Aug 29, 2016 at 3:52 PM, Evan Cordell notifications@github.com wrote:

I don't follow the bits about including delegations in pinned.json. Wouldn't those simply be in the targets.json? Are you considering the case where there are delegation roles shared across multiple repos?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-243235083, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD4xXkrqvjA7iRH3YxRGHs1O8Zr9vks5qkzhngaJpZM4Jnvng .

awwad commented 8 years ago

The addition of trust pinning has been updated in the form of a TAP here. The format of the pinned.json file has changed a bit to be similar to the normalized multi-role delegations design (which will be posted as TAP 3). Multi-repository pinning, Unix-style filename pattern matching, and terminating pinnings have also been added.

endophage commented 8 years ago

There's no way to comment directly on files in the repo so some thoughts on the TAP:

I may be missing something but the proposal seems like it won't work on a technical front. Please point out where I'm not understanding. Reading this:

    // In this example, the "Django" root metadata file specifies the timestamp
    // and snapshot keys used by PyPI.
    // However, the targets key corresponds to the Django project on PyPI, and
    // the root metadata file on disk shall never be updated.

My understanding is:

  1. that I would go to some "Django" project TUF repository to get a root.json and targets.json
  2. the root key would differ to that used for PyPI
  3. the targets key would be the same key signed into a Django delegation in PyPI
  4. the timestamp and snapshot keys would be the same as those used for PyPI

If that's all correct, where would I get a snapshot.json that references Django's root.json and targets.json. Per 4. It seems I would never actually be able to pull the TUF repository directly from Django.

In my opinion, this TAP is trying to do too much. We have a delegation mechanism in TUF already. If somebody wants to trust a different TUF repo for a different target, they should simply specify that other repo, and it should be valid in and of itself. Any sharing of files and keys across repos should be irrelevant to clients, who anchor trust only at the root.

In regards to @JustinCappos's use case of requiring a target appears in multiple repos ("under two different roots"), this seems like functionality that should be built on top of TUF but has no need to be in the spec. It's simply a case of looking up a target in two TUF repos and seeing if it matches. I don't see a particularly compelling reason to try and address every possible combination of business logic like this in the spec. People are going to come up with other use cases like this and it doesn't make sense for them all to be in the spec; Notary is already used in Docker UCP in a case where we require a target appears in multiple delegations within a single repo.

I really think it would make sense to focus on pinning just a root key for now. It may also make sense to be able to specify pinned delegation keys, and on the engineering side, a little duplication here seems like a highly acceptable trade off when compared to a high degree of complexity specifying cross repository relationships. In that respect, if the TAP specificed a pinning on root keys to repositories, and a mapping of wildcarded target names to those repositories, I think it would be awesome. The introduction of "delegated" repositories just seems unnecessary.

trishankkarthik commented 8 years ago

@endophage In case it helps, I will be updating TAP 5 shortly to make it clearer.

endophage commented 8 years ago

I was only referencing TAP 4, but a couple of comments on 5. Doesn't the spec already define a mirrors.json? Why not have a per mirror flag in there that specifies if the root.json can be updated from that mirror? What's the use case for a repo specifying a delegation as the search root? The Use Case 1 statement that a client may want to restrict its search space makes total sense, in which case, it should be specified in client configuration, not in the root.json which is global to all clients.

JustinCappos commented 8 years ago

You can safely ignore TAP 5 for now. It was more a draft from one team member, than something that had been discussed with the intent that I would approve. We will only be adding things that we think are ready to be approved to the repo in the future.

So, TAP 5 will undergo major revisions that will fix this and other issues. We'll ping you when it has been scrubbed and fixed.

On Wed, Oct 5, 2016 at 6:11 PM, David Lawrence notifications@github.com wrote:

I was only referencing TAP 4, but a couple of comments on 5. Doesn't the spec already define a mirrors.json? Why not have a per mirror flag in there that specifies if the root.json can be updated from that mirror? What's the use case for a repo specifying a delegation as the search root? The abstract statement that a client may want to restrict its search space makes total sense, in which case, it shold be specified in client configuration, not in the root.json which is global to all clients.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-251814734, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD-fH47AA1w1OuJoSFKvMLKg93x12ks5qxCCsgaJpZM4Jnvng .

trishankkarthik commented 8 years ago

@endophage

To try to answer your questions succinctly, there are two different reasons for TAPs 4 and 5. Let me try to briefly recap what they are.

TAP 4 is useful for: (1) getting different targets from different repositories (e.g., company-controlled targets first from a private repository, all other targets second from a public repository), or (2) requiring multiple repositories to sign off on the same targets.

In my opinion, this TAP is trying to do too much. We have a delegation mechanism in TUF already. If somebody wants to trust a different TUF repo for a different target, they should simply specify that other repo, and it should be valid in and of itself. Any sharing of files and keys across repos should be irrelevant to clients, who anchor trust only at the root.

To try to answer your concerns, this is exactly what TAP 4 is trying to do. Using a trust pinning file, users can specify as many repositories as they like, and exactly how they trust each repository. We think it is useful to have a standard way for users to do so.

In regards to @JustinCappos's use case of requiring a target appears in multiple repos ("under two different roots"), this seems like functionality that should be built on top of TUF but has no need to be in the spec. It's simply a case of looking up a target in two TUF repos and seeing if it matches. I don't see a particularly compelling reason to try and address every possible combination of business logic like this in the spec. People are going to come up with other use cases like this and it doesn't make sense for them all to be in the spec; Notary is already used in Docker UCP in a case where we require a target appears in multiple delegations within a single repo.

I agree that this is not a common use case. However, requiring multiple repositories (with different roots of trust) to sign off on the same targets is useful in some cases. For example, we plan to use this feature in UPTANE. It improves compromise-resilience, because you do not trust only one repository to have the final say. If two different repositories with different roots of trust (and, thus, different keys) sign the same targets metadata, then chances are better that the targets metadata is trustworthy.

In this sense, multi-repository delegations in TAP 4 is similar, but not identical to multi-role delegations in TAP 3. Multi-role delegations are also useful in improving compromise-resilience, but they still share the same root of trust. If the root keys are compromised, then the multi-role delegations can be overwritten. Not so for multi-repository delegations, because that requires compromising multiple roots of trust (which should be significantly more difficult).

Does this answer your question?

I really think it would make sense to focus on pinning just a root key for now. It may also make sense to be able to specify pinned delegation keys, and on the engineering side, a little duplication here seems like a highly acceptable trade off when compared to a high degree of complexity specifying cross repository relationships. In that respect, if the TAP specificed a pinning on root keys to repositories, and a mapping of wildcarded target names to those repositories, I think it would be awesome. The introduction of "delegated" repositories just seems unnecessary.

Precisely as you wish, TAP 5 is useful for pinning (in the root metadata file) the keys and URLs used to verify the top-level roles of a repository. There are two features in TAP 5.

First, the root metadata file can be used to control where the metadata files for top-level roles come from. If the URLs for a top-level role is not specified in the root metadata file, then the mirrors in the trust pinning file (TAP 4) would be used to download the metadata file for the top-level role. Otherwise, if the URLs have been specified in the root metadata file, then the mirrors in the trust pinning file will be ignored, and these URLs will be used instead to download the metadata file for the top-level role.

Second, the root metadata file can be used to control the keys used to verify the top-level roles. Thus, as you surmised, instead of using the root metadata file provided by a remote repository, the user can create her own metadata file to pin the keys used by this remote repository. In this way, the user need not automatically agree to revoke and replace the keys used by the remote repository. She can verify them herself before replacing the keys using her own root metadata file.

These two features allow us to implement at least one interesting use case. Suppose that a user trusts only the Django project on the PyPI repository. She is not worried about the timestamp and snapshot keys being compromised on PyPI, because the resulting attacks are not severe. However, where would she get the keys used to verify the Django project on PyPI? In the normal case, she would read the delegation to the Django project off the top-level targets role on PyPI. However, (as a purely hypothetical example), if attackers somehow compromise this top-level targets role, then attackers can distribute incorrect keys for the Django project.

If the user were so inclined, she can avoid this problem by using TAP 5. Instead of using the PyPI root metadata file, she would craft her own root metadata file. Her root metadata file would look as follows:

If that's all correct, where would I get a snapshot.json that references Django's root.json and targets.json. Per 4. It seems I would never actually be able to pull the TUF repository directly from Django.

This is a good question. Per TAP 5, the root metadata file is no longer listed in the snapshot metadata. In order to update the metadata file for a top-level role on a repository, one of the following two rules shall be used. If the URLs for the top-level role have been defined in the root metadata file, then that is where the metadata file shall be fetched. Otherwise, it shall be fetched from the mirrors defined in the trust pinning file.

So, to return to the Django example, this is how metadata files would be downloaded. First, the TUF client would either not update the root metadata file (if so instructed), or it would download the latest root metadata file on a repository of the user's control. Second, the TUF client would download the timestamp metadata file from PyPI. Third, the TUF client would download the snapshot metadata file from PyPI. Fourth, the TUF client would download the Django delegated targets role metadata file from PyPI. It would verify that the Django metadata file is signed by the keys pinned in her root metadata file, and that its version number nevertheless matches the snapshot metadata signed by PyPI. In effect, this root metadata file allows the user to restrict her trust in PyPI to provide metadata about only the Django project.

Does this make sense? If not, please let us know. I might have botched a thing or two in my explanations. If so, I apologize in advance, and I'm sure @JustinCappos can better explain things.

JustinCappos commented 8 years ago

It looks like an accurate (and apt) summary. Thanks!

On Thu, Oct 6, 2016 at 4:47 AM, Trishank Karthik Kuppusamy < notifications@github.com> wrote:

@endophage https://github.com/endophage

To try to answer your questions succinctly, there are two different reasons for TAPs 4 and 5. Let me try to briefly recap what they are.

TAP 4 https://github.com/theupdateframework/taps/blob/master/tap4.md is useful for: (1) getting different targets from different repositories (e.g., company-controlled targets first from a private repository, all other targets second from a public repository), or (2) requiring multiple repositories to sign off on the same targets.

In my opinion, this TAP is trying to do too much. We have a delegation mechanism in TUF already. If somebody wants to trust a different TUF repo for a different target, they should simply specify that other repo, and it should be valid in and of itself. Any sharing of files and keys across repos should be irrelevant to clients, who anchor trust only at the root.

To try to answer your concerns, this is exactly what TAP 4 is trying to do. Using a trust pinning file, users can specify as many repositories as they like, and exactly how they trust each repository. We think it is useful to have a standard way for users to do so.

In regards to @JustinCappos https://github.com/JustinCappos's use case of requiring a target appears in multiple repos ("under two different roots"), this seems like functionality that should be built on top of TUF but has no need to be in the spec. It's simply a case of looking up a target in two TUF repos and seeing if it matches. I don't see a particularly compelling reason to try and address every possible combination of business logic like this in the spec. People are going to come up with other use cases like this and it doesn't make sense for them all to be in the spec; Notary is already used in Docker UCP in a case where we require a target appears in multiple delegations within a single repo.

I agree that this is not a common use case. However, requiring multiple repositories (with different roots of trust) to sign off on the same targets is useful in some cases. For example, we plan to use this feature in UPTANE https://uptane.github.io. It improves compromise-resilience, because you do not trust only one repository to have the final say. If two different repositories with different roots of trust (and, thus, different keys) sign the same targets metadata, then chances are better that the targets metadata is trustworthy.

In this sense, multi-repository delegations in TAP 4 is similar, but not identical to multi-role delegations in TAP 3 https://github.com/theupdateframework/taps/blob/master/tap3.md. Multi-role delegations are also useful in improving compromise-resilience, but they still share the same root of trust. If the root keys are compromised, then the multi-role delegations can be overwritten. Not so for multi-repository delegations, because that requires compromising multiple roots of trust (which should be significantly more difficult).

Does this answer your question?

I really think it would make sense to focus on pinning just a root key for now. It may also make sense to be able to specify pinned delegation keys, and on the engineering side, a little duplication here seems like a highly acceptable trade off when compared to a high degree of complexity specifying cross repository relationships. In that respect, if the TAP specificed a pinning on root keys to repositories, and a mapping of wildcarded target names to those repositories, I think it would be awesome. The introduction of "delegated" repositories just seems unnecessary.

Precisely as you wish, TAP 5 https://github.com/theupdateframework/taps/blob/master/tap5.md is useful for pinning (in the root metadata file) the keys and URLs used to verify the top-level roles of a repository. There are two features in TAP 5.

First, the root metadata file can be used to control where the metadata files for top-level roles come from. If the URLs for a top-level role is not specified in the root metadata file, then the mirrors in the trust pinning file (TAP 4) would be used to download the metadata file for the top-level role. Otherwise, if the URLs have been specified in the root metadata file, then the mirrors in the trust pinning file will be ignored, and these URLs will be used instead to download the metadata file for the top-level role.

Second, the root metadata file can be used to control the keys used to verify the top-level roles. Thus, as you surmised, instead of using the root metadata file provided by a remote repository, the user can create her own metadata file to pin the keys used by this remote repository. In this way, the user need not automatically agree to revoke and replace the keys used by the remote repository. She can verify them herself before replacing the keys using her own root metadata file.

These two features allow us to implement at least one interesting use case. Suppose that a user trusts only the Django project on the PyPI repository. She is not worried about the timestamp and snapshot keys being compromised on PyPI, because the resulting attacks are not severe. However, where would she get the keys used to verify the Django project on PyPI? In the normal case, she would read the delegation to the Django project off the top-level targets role on PyPI. However, (as a purely hypothetical example), if attackers somehow compromise this top-level targets role, then attackers can distribute incorrect keys for the Django project.

If the user were so inclined, she can avoid this problem by using TAP 5. Instead of using the PyPI root metadata file, she would craft her own root metadata file. Her root metadata file would look as follows:

  • root URLs: either an empty string (to denote "never update this root metadata file"), or to a file on a repository of her own control
  • root keys: her own root keys
  • targets URLs: the URL to the Django project (i.e., delegated targets role) on PyPI
  • targets keys: the Django targets keys, obtained through an out-of-band mechanism
  • snapshot URLs: the URL to the snapshot metadata file on PyPI
  • snapshot keys: the keys to the snapshot role on PyPI
  • timestamp URLs: the URL to the timestamp metadata file on PyPI
  • timestamp keys: the keys to the timestamp role on PyPI

If that's all correct, where would I get a snapshot.json that references Django's root.json and targets.json. Per 4. It seems I would never actually be able to pull the TUF repository directly from Django.

This is a good question. Per TAP 5, the root metadata file is no longer listed in the snapshot metadata. In order to update the metadata file for a top-level role on a repository, one of the following two rules shall be used. If the URLs for the top-level role have been defined in the root metadata file, then that is where the metadata file shall be fetched. Otherwise, it shall be fetched from the mirrors defined in the trust pinning file.

So, to return to the Django example, this is how metadata files would be downloaded. First, the TUF client would either not update the root metadata file (if so instructed), or it would download the latest root metadata file on a repository of the user's control. Second, the TUF client would download the timestamp metadata file from PyPI. Third, the TUF client would download the snapshot metadata file from PyPI. Fourth, the TUF client would download the Django delegated targets role metadata file from PyPI. It would verify that the Django metadata file is signed by the keys pinned in her root metadata file, and that its version number nevertheless matches the snapshot metadata signed by PyPI. In effect, this root metadata file allows the user to restrict her trust in PyPI to provide metadata about only the Django project.

Does this make sense? If not, please let us know. I might have botched a thing or two in my explanations. If so, I apologize in advance, and I'm sure @JustinCappos https://github.com/JustinCappos can better explain things.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-251902688, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XDyeUPXKWiu0ETQdhFdRDYmUTOtDEks5qxLWNgaJpZM4Jnvng .

endophage commented 8 years ago

@trishankkarthik thanks for the long reply, I think I understand what is being suggested per the Django example now. I'll have to spend some time thinking about it because it still appears to be far more complex than necessary, but there may be some complexities I haven't fully grokked. It seems the goal is to provide some degree of control over a repository owner (i.e. PyPI) arbitrarily changing delegation keys of publisher (i.e. Django). That is definitely a worthwhile feature that I've been wondering about so I'm happy to see some thoughts along that line.

The remainder of the discussion is largely me asking: why is it be desirable to include business logic implementations in the TUF spec? Without any change to the spec they are implementable as needed. On the other hand, adding a specific business case to the spec may conflict with other use cases. Multi-repo confirmation of a target is an application feature, not a specification feature. I guess by way of example, I'd say the TUF spec is to your UPTANE use case, what x509 (RFC 5280) is to Certificate Transparency (RFC 6962). Maybe it would be similarly desirable to have "Target Transparency" and "Key Transparency" specs that sit on top of TUF, without requiring it to be modified.

trishankkarthik commented 8 years ago

On 6 October 2016 at 13:05, David Lawrence notifications@github.com wrote:

The remainder of the discussion is largely me asking: why is it be desirable to include business logic implementations in the TUF spec? Without any change to the spec they are implementable as needed. On the other hand, adding a specific business case to the spec may conflict with other use cases. Multi-repo confirmation of a target is an application feature, not a specification feature. I guess by way of example, I'd say the TUF spec is to your UPTANE use case, what x509 (RFC 5280) is to Certificate Transparency (RFC 6962). Maybe it would be similarly desirable to have "Target Transparency" and "Key Transparency" specs that sit on top of TUF, without requiring it to be modified.

Glad to hear the summary helped!

Let me try to assuage your concern by saying that TAP 4 does not change at all the most common use case of using a single repository for all targets. That popular use case will continue to work as is, requiring only a trust pinning file to delegate all targets to the repository.

I'll let @JustinCappos address the rest of your concerns :)

JustinCappos commented 8 years ago

The remainder of the discussion is largely me asking: why is it be desirable to include business logic implementations in the TUF spec? Without any change to the spec they are implementable as needed. On the other hand, adding a specific business case to the spec may conflict with other use cases. Multi-repo confirmation of a target is an application feature, not a specification feature. I guess by way of example, I'd say the TUF spec is to your UPTANE use case, what x509 (RFC 5280) is to Certificate Transparency (RFC 6962). Maybe it would be similarly desirable to have "Target Transparency" and "Key Transparency" specs that sit on top of TUF, without requiring it to be modified.

So, from my perspective, TAP 4 isn't a business logic implementation. This is functionality that is common to a large number of TUF use cases, it is core update functionality, and it is easy to get it wrong from a security standpoint.

I certainly wouldn't want TUF to prohibit using some existing technique like CT. However, in vanilla TUF (with CT), how do you handle the case of pinning Django's key? I believe what the user expects is that this means that the other roles on that repo could be compromised but Django packages will be retrieved using Django's key. It will not work that way without changes (similar to those in TAP 4) that make the namespace start with the pinned role.

I don't see TAP 4 as being much more than the minimum one would need to support CT in a sane way. Let's discuss if there is something that we could easily remove while keeping the mechanism general.

Thanks, Justin

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/tuf/issues/361#issuecomment-252026245, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD_mVaHZMQOMQ8UcveIpDj_oyI7FQks5qxSpXgaJpZM4Jnvng .

ecordell commented 8 years ago

Just to chime in here, I also don't see TAP4 as being "business logic." It certainly enables some other things, but fundamentally TAP4 is all about "where to I go to get these packages, which signatures should I trust, and why", which is exactly what TUF is all about.

Certainly you could implement something similar on top of TUF; but I would argue that anything involving decisions about which TUF repository/repositories to consider (based on target name) is encoded by TAP4. TAP4 lets you express arbitrary order and number of repositories that must include a target. Since this changes the entrypoint into TUF, it's an important thing to consider carefully from a security standpoint and is a good candidate for standardizing. (e.g. what are the security implications of making pinned.json optional?)

ecordell commented 7 years ago

Closing since this is now represented by TAP 4.