executors: Native uploadstore support for artifacts

[eseliger]

Today, both executor queues (codeintel & batches) push an artifact to Sourcegraph as the result of their execution.

• For batch changes, that are the step results, currently encoded in the logs.
• For Code Intel, that is the LSIF/SCIP dump, uploaded via a route open to the executor.

It would be great to have an abstraction here and to instead allow to provide a pattern of files that will be uploaded at the end of the execution to the blob store. These will then be available in the OnCompletion handler on the server side for further processing.

Requires a working generic upload store implementation.

The ultimate goal of this is to reduce required special endpoints for executors that are non-generic and to break SSBCs dependency on keeping logs forever. Currently, the UI would break if the logs are gone.

[eseliger]

Note: batch changes today relies on these results to be streamed. This would need to be accounted for, or we will at least still need to read those from the logs before the job completes. Ideally this will drastically reduce the size of the logs as well.

[eseliger]

Steps:

• Prepare for having a bucket available for batch changes
• Add an Artifacts []{Path string; Bucket string} field to the DockerStep executor Job interface type
• Extract the logic for uploading to an upload store from src-cli for codeintel into lib/uploader
• Reuse that logic in the src-cli package to reduce duplication
• Implement logic in the executor to find an artifact if given in the step definition, use that uploader package to do the upload
• Implement an upload store and uploadStore endpoints for the batches bucket
• Mount a special route set under the executorqueue router to authenticate uploads using the executor secret
• Add store methods to get the uploads for an execution job
• Overwrite the OnComplete handler on the queuestore for codeintel like we do for batch changes to do some post-processing required on the server side
  • For that handler we need to
  • Extract metadata from the index
  • Then build the LsifUpload record and insert it into the DB so upload processing starts (this logic currently lives in src-cli and needs to be ported to the backend)
• For batch changes, read the AfterStepResults from the bucket instead of parsing the json logs
• Remove the special endpoint for uploading LSIF data from the executorqueue router
• Remove passing down the executor queue secret to the codeintel lsif upload step

Done ✨