It would be really nice to have a collection of test data to test client conformance

[patflynn]

Generating test data for TUF clients is time consuming and error prone. It could be really cool to have a test data set to exercise clients and certify that they're implementing all elements of the update algorithm.

[znewman01]

Glad you filed this, it comes up over and over and AFAIK there's no place to track.

There's at least one abortive attempt at this along with semi-related prior art. It also comes up pretty often in the go-tuf meeting (see 2022-08-03 meeting, for instance) and the TUF community meeting (see 2022-07-27, 2021-08-25 meetings).

There are two related suites of tests:

1. Conformance tests: we should be able to write these based solely on the spec
2. Interoperability tests: each client implementation should be able to consume the output of each other repository implementation (test matrix)

There are a couple of issues that make this nontrivial (but still very worth doing!):

• The different libraries present very diverse APIs, and don't agree on e.g. the primitive operations that a client would do. It's hard to (doubly so for the interop tests, when we want each client to generate a repo according to a template).
  • This makes it hard to create e.g. a YAML doc that describes various test inputs/outputs.
• There's a fair amount of flexibility permitted in the client implementations. For instance, the POUF system means that, strictly speaking, a client implementation doesn't need to understand JSON to be conformant.
• There's a huge variety of scenarios to test. It might be worthwhile to use a fuzzing approach.

[asraa]

Yes!! We were talking about this together in the Sigstore world and definitely think it's useful to start with the Sigstore TUF clients. There's a starter conformence tests for Sigstore clients that we can piggy-back off of for the TUF root.

• Diverse APIs: We can probably dictate with Sigstore TUF clients on their API
• All sigstore tuf client implementations MUST be able to understand the Sigstore TUF repository
• We definitely should catch corner cases as we update the Sigstore TUF client for more data

[joshuagl]

Zack already linked to the description of how we handle testing in python-tuf. I also want to point out the repository-editor-for-tuf, which we've found to be a useful tool for generating metadata and reviewing the changes to the metadata over time/TUF operations (because repository-editor-for-tuf uses a git repository for storage).

[patflynn]

@joshuagl that repository editor tool looks fantastic thanks for the pointer.

I think the differences in API aren't a huge deal. If we have a solid suite of test data it should be trivial for each client to implement a test driver for their API.

[znewman01]

I think the differences in API aren't a huge deal. If we have a solid suite of test data it should be trivial for each client to implement a test driver for their API.

I wish I agreed with you :(

Some of the things the libraries disagree about include whether it's possible to point at an on-disk/in-memory TUF repo, whether it's possible to iterate targets metadata, whether you can access metadata at all or you just get a "download file" primitive, etc.

I think/hope in the medium term that there will be some convergence here, and perhaps such a test suite can drive convergence.

[asraa]

I'm going to start a high-level doc about this this week

Some of the things the libraries disagree about include whether it's possible to point at an on-disk/in-memory TUF repo, whether it's possible to iterate targets metadata, whether you can access metadata at all or you just get a "download file" primitive, etc.

At minimum, ALL need to point to a remote TUF repo. If clients want additional testing against the on-disk/in-memory, that shouldn't be that hard to configure given a test suite.

whether it's possible to iterate targets metadata

We know now that at minimum all TUF clients implement the GetTarget(filename) API to retrieve the metadata about the target and DownloadTarget(filename) to get the filecontent. Given this, we're working on a target discovery API (if a repo provides a well known index.json file, it can provide the names of the rest of the targets).

That's all that should be relied on. On a further Sigstore client layer, we should rely on the TrustedRoot protobuf to be produced given a remote URL defined in protobuf-specs. For now, we can just start with the TUF functionality, and implement conformence for updating metadata and retrieving targets given the filename.

whether you can access metadata at all or you just get a "download file" primitive, etc.

I think all TUF clients can download the metadata about the target before calling a download on the target file name to get the file content.