Provide docs/standards about client/worker/proxy interactions in redeployable taskcluster

[petemoore]

Changes are required for clients of taskcluster services when taskcluster becomes redeployable.

This RFC serves to define the standards by which taskcluster deployments will publish a manifest of their service definitions, how client generators will query and interpret this manifest and associated reference and schema documents in order to generate clients, what the architectural changes to generated clients will be, how users of those clients (workers, worker authentication proxies, command line tools, software libraries) will be adapted in order to use the new clients, and how generated clients and client generators will be built, released and deployed.

This entails changes to the way workers will share deployment / authentication proxy configuration information with tasks, versioning of the API / Event reference schemas, versioning of the services definitions based on those API and Event reference schemas, and the possibility for clients to connect to different taskcluster deployments.

[djmitche]

Is this going to include schemas for reference files and so on (like https://bugzilla.mozilla.org/show_bug.cgi?id=1476602)? If so, I'll stop working on that until this RFC is complete.

[petemoore]

Regarding the renaming (api -> http, events/exchanges -> amqp), I don't feel like the new names make more sense than the old, but the old aren't perfect. It'd be nice to be consistent, whatever we choose, but the fewer things we rename, the easier this work will be. Renaming always seems easy, then you end up breaking things in production because you missed somewhere!

That is a fair point. And I agree the new names aren't considerably better, so I will go back to the old names. The main naming issue I was trying to solve was that we have no collective term for API meaning HTTP API or AMQP API. We use API for the collective term, but then also sometimes just meaning the HTTP API (e.g. in the name apis.json). For example, does API references mean something that adheres to api-reference.json or exchange-reference.json, or just the former?

Maybe I can just update the docs to make it clear in each place I reference these terms, what I mean...

[petemoore]

Note, I'll squash commits when I land, of course, and give a reasonable commit message!

This is ready for review, I am not intending to land any more changes.

[djmitche]

So, the meaning of the env vars section remains ambiguous, and there appears to have been no final-comment period here in which to catch that issue or check that release engineering and other client consumers are OK with the change. I don't want to delay this by standing on ceremony, and I think we have consensus on most of this, but I'm worried that we've missed discussion of a consumer-facing breaking change. Maybe we should consider pulling out the env-var bit into a separate RFC?

[petemoore]

there appears to have been no final-comment period here in which to catch that issue or check that release engineering and other client consumers are OK with the change

Apologies, I forgot about applying the tags, but I think we have a general consensus based on the review approvals. It is always plausible for there to be slight changes during the implementation phase, but I think as long as we are vocal about changes coming, and communicate these things well, we should be ok. When users upgrade a major version, they are generally aware there can be breaking changes - the mistake we have made in the past is not being very vocal about it and explicit in release notes. As long as the release notes are very explicit about the breaking changes, and we communicate them well, users can decide if they wish to upgrade, and how long they need for adjusting to the changes. I think this is where we have fallen down in the past.