So how do I verify that everything worked?

[aaaaalbert]

I finished the wizard, downloaded my layout, and instructed my functionaries to change their build process --- but how do I verify that in-toto did actually work its magic?

[SantiagoTorres]

Just trying to contextualize this question. Does this mean that you'd want to add documentation on how to verify the layout? or something else?

[aaaaalbert]

I expected at least a very simple "pastie" that lets me verify the metadata that my functionaries create, the way my end users would do this.

The subsection "Instruct your Functionaries to use in-toto Commands" at https://in-toto.engineering.nyu.edu/wrap-up is very specific and shows exactly what every functionary should do now to create the metadata. I'm missing a subsection à la "This is how an end user will check their downloaded software for in-toto compliance" (WIP title) that explains exactly this part.

(A current-day analogy would be how a developer calculates and publishes a hash/signature for a downloadable file; the end user re-calculates the hash/sig for their downloaded copy and compares it with the published one. Sure the dev can and should check that the step to be performed by the user actually yield the correct result!)

On a side note: "Release in-toto verifiable Software" tells me to ship in-toto link metadata file(s) with my product, an interesting task in itself as I've packaged the software already.

[SantiagoTorres]

I expected at least a very simple "pastie" that lets me verify the metadata that my functionaries create, the way my end users would do this.

Ideally, you shouldn't be instructing users on how to do it, else they are likely not to do it (see: your current day analogy). We are targeting to integrate with package managers and users tool to do this under the hood. As they say, no news, good news.

We can add a snippet on how to run in-toto verify, if that's the concern, although I'm not sure if at this point it would change too much. Adding to it, there is many ways to verify the hash of a file once it's downlloaded (e.g., sha256sum on my terminal, or using a GUI tool that's part of the DE).

"Release in-toto verifiable Software" tells me to ship in-toto link metadata file(s) with my product, an interesting task in itself as I've packaged the software already.

I'm assuming that's to be defined by the package manager/tool-that-verifies in-toto. I don't think it necessarily has to be stick in the same tarball. As an analogy, TUF, metadata is not part of the target files, but rather something that is shipped in some way to verify the target files.

Would it help to add this information somewhere to clear things up?

[aaaaalbert]

Thanks for your reply!

I expected at least a very simple "pastie" that lets me verify the metadata that my functionaries create, the way my end users would do this.

Ideally, you shouldn't be instructing users on how to do it, else they are likely not to do it (see: your current day analogy). We are targeting to integrate with package managers and users tool to do this under the hood. As they say, no news, good news.

OK, I just learned the bit about planned "background" verification that requires no interaction from the user. I obviously thought I get something like a signature out of the in-toto'ed build process, and I expected verification steps for it.

So let's substitue "end user" with "end user's package manager" in my original quote. How can I, as the developer that uses the web wizard, arrive at the same conclusion as the package manager that checks whether my software passes in-toto verification?

What I have is metadata from the wizard, detailed steps to create yet more / different metadata, and I know I should ship this with my software.

I guess I will need to run a full build (and QA and packaging) using the prescribed in-toto commands, and then verify the outcomes of these steps. The wizard tells me the in-toto commands to use for the build etc., but what about the verification step?

We can add a snippet on how to run in-toto verify, if that's the concern, although I'm not sure if at this point it would change too much. Adding to it, there is many ways to verify the hash of a file once it's downlloaded (e.g., sha256sum on my terminal, or using a GUI tool that's part of the DE).

If in-toto-verify does what I tried to sketch, then great, make the wizard tell me about it!

I'm not sure what to make of "many ways to verify a hash". My goal sure isn't to redo all verification steps by hand.

"Release in-toto verifiable Software" tells me to ship in-toto link metadata file(s) with my product, an interesting task in itself as I've packaged the software already.

I'm assuming that's to be defined by the package manager/tool-that-verifies in-toto. I don't think it necessarily has to be stick in the same tarball. As an analogy, TUF, metadata is not part of the target files, but rather something that is shipped in some way to verify the target files.

Got it, no worries about that then.

Would it help to add this information somewhere to clear things up?

Perhaps there's a clearer way to explain how metadata and the actual software build then form another sort of package, like for a package manager.

[SantiagoTorres]

I guess I will need to run a full build (and QA and packaging) using the prescribed in-toto commands, and then verify the outcomes of these steps. The wizard tells me the in-toto commands to use for the build etc., but what about the verification step?

If in-toto-verify does what I tried to sketch, then great, make the wizard tell me about it!

Right, I see where you're coming from. I was just trying to gauge whether we were talking about "this should be said somewhere in the site" rather than "I don't understand this" (a third option is "I don't understand this, you should put it in your site").

I think for the sake of clarity we can add a paragraph somewhere in the command listing with something akin to

Your distribution media (e.g., package manager) may handle and verify in-toto metadata in the future, in the meantime, you can ask your users to do the following:

     [block with the in-toto-verify command]

In  order to verify the integrity of their supply chain

@aaaaalbert Is this what you had in mind? @lukpueh what's your take on it?

[aaaaalbert]

I don't care so much about the future actually, I as the developer want to verify that I and the tool/commands did what we are supposed to do.

I'd go for a simpler "To verify the in-toto metadata manually, do the following." This sort of indicates that automation may exist. I don't think the end user needs to be mentioned for this kind of dev's sanity check.

It's sure a matter of me not understanding in-toto deeply enough. Perhaps the intro page could help adjust the wizard user's mindset by explaining the process and the outcomes? I developed my own expectations as I went along, and many of them turn out not quite right 😄.

[SantiagoTorres]

I'd go for a simpler "To verify the in-toto metadata manually, do the following." This sort of indicates that automation may exist. I don't think the end user needs to be mentioned for this kind of dev's sanity check.

++

It's sure a matter of me not understanding in-toto deeply enough. Perhaps the intro page could help adjust the wizard user's mindset by explaining the process and the outcomes? I developed my own expectations as I went along, and many of them turn out not quite right 😄.

It's completely understandable. I think as time passes, we'll be doing a better job at getting people in the right mindset before they infer the wrong things.

Thanks for your feedback!

[aaaaalbert]

Sure, happy to help!

[lukpueh]

I'd say a demonstration of the verification process would definitely help users of this tool (whoever that might be) to better understand how in-toto works and more importantly allow them to correct mistakes and improve their layouts.

Even as this tool evolves and as we do a better job at getting people in the right mindset, it will probably remain easy for users to create faulty or invalid layouts that inadequately reflect the related real world supply chains.

I am convinced that a demonstration of a (partial) verification could eliminate some misconceptions of the system or prevent mistakes that might occur even among in-toto-savvy users.

Btw. we still haven't fully agreed on what the guarantees page should look like, but IMO its contents are closely related to a preliminary verification as suggested here.