RFC: Optionally sign public S3 URLs

[ricky26]

An RFC for optionally signing public S3 URLs, so that the public artefact bucket can be not globally readable.

[ricky26]

@petemoore I completely agree with this disappearing down the line if we can come up with a workable solution for private logging. To be honest, that combined with flexible declarations of where assets go would be ideal (as we briefly discussed the other day).

Note, in other documentation we tend to use "artifact" and "Taskcluster" rather than "artefact" and "TaskCluster" but this is a pretty insignificant nit.

Aha, thanks, I didn't really think about the spelling of artifact (and I somehow didn't absorb how you capitalise Taskcluster). Thanks for pointing it out. 😄

[djmitche]

@escapewindow, you've done a good bit of thinking about artifacts. Any thoughts here?

[escapewindow]

Hm. I'm wondering if we want to sign public artifact urls, or if we want to configure a cluster to have no public artifact urls. I'm thinking something like:

FirefoxCI:
publicArtifactPrefixes: ["public/"]

PrivateCluster:
publicArtifactPrefixes: []

[escapewindow]

(We might want to also configure the default location of task artifacts and logs so we don't end up with public/logs/live.log if it's actually a private artifact in that cluster.)

[ricky26]

@escapewindow I wanted to propose something a little more flexible too but it sounded like it was something people had already thought about and would need a bigger discussion. The hope here is that this is a minimalistic change which unblocks private clusters. Ideally, this would be removed at some point with a proposal for more flexible artifact storage.

Defining public artifact prefixes (or regular expressions) sounds like a good step up but I'm worried that it might be hard to get it right without causing compatibility issues, since I think more areas make assumptions about the public/ prefix. At the moment, I don't think it's really practical for other users to care at all about what format the redirect URL takes, so it seems safer. Any areas which cared about the public/ prefix would have to be revisited anyway if we reworked how artifact storage was implemented.

However, I'm no expert in these areas. 😅

[escapewindow]

As noted elsewhere, I'm good with changes that don't break FirefoxCI and help make Taskcluster usable in a private cluster. Sounds like we're in agreement that we can revisit how we implement private clusters later. 👍

[petemoore]

Cool, it sounds like everyone is happy, let's go with this!

[djmitche]

Per mechanics.md this shouldn't have been merged yet, so let's pretend it wasn't!

The next step is a final comment period. I'll send a note to tools-taskcluster drawing attention here. Please feel welcome to make comments despite the purple "merged" chip. If we need to modify the proposal, we can do so in another PR (keeping the RFC number).

[petemoore]

Whoops, sorry, my bad! I forgot about the process, will make sure I don't do that again...

[ccooper]

The Final Comment period will end on Monday, September 28.