Closed davedoesdev closed 8 years ago
FWIW, I have some personal notes which I could clean up and convert to Markdown and submit as a PR—if there is a general consensus that the protocol documentation will be kept up to date; having obsolete/wrong documentation would probably be worse than having no documentation.
(For a starting point, see server/server.go:RootHandler
. )
@mtrmac thanks. As an alternative I was thinking of using TUF and putting the metadata files into the Docker registry itself. I'm wondering what the downsides to doing that are?
@davedoesdev Probably possible, but nothing in the registry, the Docker client, or the notary CLI implements storing TUF files into the registry. Am I just misunderstanding you?
Specific downsides:
@mtrmac Not misunderstanding. I'm hoping to use https://github.com/davedoesdev/dxf to store TUF files in the registry. I'm planning to use the TUF Python implementation to manage the metadata and upload it using dxf. The targets will be manifests which point to the content (using their hashes).
Work-in-progress here: https://github.com/davedoesdev/dtuf - any feedback gratefully received, I'm working on the docs so I realise it may be a bit impenetrable. Regarding the downsides:
@davedoesdev @mtrmac sorry this issue got ignored while we were working on the hardware signing leading up to DockerCon.
Any help with docs would be greatly appreciated. Long term, we really don't want to maintain HTTP API docs, they're too prone to being wrong. We'd like to put a GRPC interface in place (c.f. https://github.com/docker/notary/issues/14) and have that be the officially supported external interface to the Notary server.
This has been something we've discussed and up to now rejected. To make parallel updates to the same TUF repo safe in a high availability environment, a distributed locking mechanism is required to ensure updates to individual files don't get interleaved, creating an invalid repo. Even with consistent snapshotting in TUF, a user could push something and not see their change show up because somebody else's change beat it out. Using a transactional data store allows us to do safe atomic updates of the entire repository.
Currently the TUF server does a lot of validation to ensure that an upload will not leave a repository in an inconsistent state that would prevent a client from successfully pulling it. This kind of validation is something I'm pushing for more of in the registry. I frankly think it's an appalling experience to permit user to push something with no errors, then subsequently be unable to pull it because it fails to validate.
Adding support for consistent snapshots will make this much easier. A timestamp can be requested at a specific version then there is a natual way to include the hashes of other version of files (which will be versioned via the timestamp and TUF merkel tree structure).
@endophage thanks for your reply, appreciate it. An RPC interface would be much better, I'll close this issue in favour of #14. I'll continue to progress storing TUF data in the registry. It's working, including consistent snapshots, in https://github.com/davedoesdev/dtuf, but I have the unit tests and docs to tidy up. When notary has an RPC interface, I'll look at switching.
Just a quick question regarding safe atomic updates. Do you mean that if one's update has been superceded while pushing, you get an error?
The issue with atomic updates will mostly occur if 2 updates coming in update an overlapping set of TUF files, and due to not having an atomic update against the backend storage, a subset of each update gets selected. As an example:
With consistent snapshots, this won't be a problem as far as integrity. The timestamp will point at SnapshotB and via using hash references a client will get TargetsB. However the user that just pushed update C will not see their updates.
Using a database with a composite key, we currently update (TargetsB, SnapshotB) as a single atomic update, and a composite key that includes the version ensures that when we try and apply (TargetsC, SnapshotC) the key constraint causes an update failue (because B and C are both at Version A+1). In this instance the author of Update C is at least messaged and can retry their update, applying it on top of B.
@endophage got it - thanks again. I can live with this for now as only I'll be updating. I'll keep an eye out for that RPC interface :-)
The README says:
Is the HTTP API documented somewhere?