FR: support Kurtosis packages in private repositories

[leeederek]

Background & motivation

This feature request captures the need to support Kurtosis packages published to private Github repositories. This may be a blocker for some folks.

Desired behaviour

I can run Kurtosis packages that are published to private Github repositories.

How important is this to you?

Painful; the lack of this feature makes using Kurtosis frictionful.

[mieubrisse]

Did some design-thinking about this last night. In order to access private repos, we need a Github read token that allows us to get access to the private repo. And specifically, the APIC needs a secret (Github token or SSH key) for accessing the repo.

However, we of course only want the user to need to give access once, which means that we'd want to store the token in the engine (and then pass the token to all created APICs so they can get access).

So, my suggestion:

• The CLI has some command that allows the user to upload the secret (likely a token) to the engine
• When the user calls this command, we should guide them through the process of creating a read-only token for accessing the repos
• For a v0, this should happen through an endpoint on the engine, NOT through environment variables or anything. Rationale: that way, the engine can store the secret on disk and have that secret even through engine restarts.
• This of course requires the engine to have a database, and it should be encrypted at rest for maximum security.
• But, to encrypt the database at rest, we need a decryption password that's provided when the engine restarts. This adds a whole extra layer of complexity to starting the engine, because then we need to generate the decryption key, store it somewhere, make sure the user doesn't lose it, etc.
• For a local Kurtosis installation (on Docker or Minikube), we might be able to store the secrets as a file on the user's local filesystem and then bind-mount the file into place in the engine
• In the Cloud though, we can't do this because anybody could get access, so we should use a secrets store - something like Hashicorp Vault or Infisical - rather than trying to write our own database encryption/decryption logic.

The above is a lot of work, but IF we do this then we get a couple neat things for free:

• Access to private repos (this ticket)
• Access to private Docker registries
• Potentially user-defined secrets in Starlark (sensitive values or parameters being fed into Starlark)

[leeederek]

Current use cases from Ava Labs and Merkle.io are for users who want Kurtosis to be installed & used to spin up environments as part of CI. Currently, if a user does this in their CI environment, then private images will work fine because the CI environment has access to the image tags required. Bumping this down a bit in priority because we have workarounds for the known use cases we've encountered so far.