Decide what to do with non-versioned metadata

[jku]

tuf-on-ci does not currently publish any non-versioned metadata apart from timestamp.json. The repository has enabled "consistent snapshot" quite a long time ago so I think this is not problematic: clients should not be using non-versioned metadata.

However:

• The GCS bucket still contains the old files
• so (non-versioned) root.json does not actually contain the newest root

This is not ideal so let's make a decision:

• either we stop supporting non-versioned metadata and remove them from GCS bucket
• or we add a bit of code complexity somewhere to support old test scripts scripts (or badly configured clients)

[haydentherapper]

either we stop supporting non-versioned metadata and remove them from GCS bucket

Any downside to this? If no, this seems reasonable.

[jku]

Real clients should use consistent snapshot so should never touch non-versioned metadata. I'm not aware of any client operating differently.

I imagine there are testing scripts etc somewhere that may download "$URL/root.json" to initialize a client... but those should be easily fixable.

My gut feeling is that keeping things simple is worth it: we should only deploy files that we actually care about and test (and should not deploy non-versioned files). The "saved" complexity is not much but I think it all adds up.

[joshuagl]

Per 5.3.3 of the detailed client workflow all clients should retrieve root with a versioned prefix, so non-versioned root.json is a bit fishy regardless. IIRC some TUF implementations would write the latest version of metadata with a non-versioned prefix, even if consistent snapshots were enabled. It's debatable whether these should ever have been published.

I agree that only files which are used and are tested should be published and non-versioned metadata (other than timestamp.json, which is always non-versioned) should be removed.

[jku]

• On staging there are almost zero requests for non-versioned metadata
  • sigstore-probers is the main one: I'll update that soon
• On production unfortunately there's quite a bit of use of non-versioned metadata
  • Easily > 95% of these requests are clearly not from TUF clients... but I don't have an automated way to verify that at this point

I still think there shouldn't be real clients using non-versioned metadata and we should bite the bullet here and remove the non-versioned data

[jku]

• On production unfortunately there's quite a bit of use of non-versioned metadata

  • Easily > 95% of these requests are clearly not from TUF clients... but I don't have an automated way to verify that at this point

Actually I take it back: this seems to be a strange failure mode in cosign: it downloads non-versioned metadata "for debugging purposes" if the actual metadata fails to verify... Since old cosign releases fail with current metadata, they end up in this failure mode.

[jku]

For staging it turns out that fixing sigstore-probers and gitsign was enough: there are now no users of non-versioned staging metadata.

I will try to remove root.json, snapshot.json, targets.json and registry.npmjs.org.json on GCS now -- I assume I don't have the permissions though.

EDIT: Indeed I don't have the permissions. To reiterate the required steps:

• I want to delete non-versioned metadata in tuf-root-staging GCS bucket (in "Sigstore staging" project)
• timestamp.json is required and should not be deleted
• root.json, snapshot.json, targets.json and registry.npmjs.org.json should be deleted

[haydentherapper]

@jku I've deleted root.json, snapshot.json, targets.json and registry.npmjs.org.json from tuf-root-staging

[jku]

Thanks! I'll close this one and add a bullet point to my "differences between staging and prod" list