Support private git repositories when building software environments

[mrocklin]

Coiled currently allows users to include public git repositories in their software environments by including them in their pip= requirements

coiled.create_software_environment(
    ..., 
    pip=["git+https://github.com/dask/dask@master"],
)

However, sometimes users have software in private git repositories. Is there a convenient way to support these?

[mrocklin]

Today the answer is "no" for a couple of reasons:

1. We don't yet collect and manage git credentials (although this would not be hard to implement if there are a few requests here)
2. Currently everything is open-access, so other folks would be able to run your software. We can add in private accounts and private software (we some infrastructure for this already for paid accounts) but haven't yet prioritized it for Coiled's public service

As with all things though, this isn't hard to build, it's just a matter of prioritization. If this is important to folks I encourage them to :+1: this issue.

[jose-moralez]

We have some software that is in private repositories on github and our use case would involve creating a software environment from these repos and keeping it private. I like the git credentials option, another option that would probably work for us would be to be able to provide an url to a docker image in a private registry such as ECR and provide some kind of authentication.

[mrocklin]

They're both sensible approaches. We'll probably enable both in the future. The easiest things for us to do in order are ...

1. Optionally expose privacy settings to team accounts
2. Let you specify a git ssh key for an account, and use that ssh key when building docker images
3. Let you specify an IAM role for us to use to build and launch things in your account

We'll end up doing all three eventually, but 1 and 2 are moderately easy to do soon. cc @marcosmoyano for 1, 2 and @dantheman39 for 2 I think.

[mrocklin]

Checking in here @jose-moralez . I thought that you'd like to know that this is in progress.

Also @marcosmoyano if you wanted to walk through your proposed solution with @jose-moralez , either here or in a call, it might be good to get feedback.

[marcosmoyano]

Sure thing! My proposed solution is as follows:

• Users who need access to a private repo should set up a personal access token (https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token)
• Add the token to their Coiled user
• When creating a software environment, the format to support pip private repos would be something like git+https://GIT_TOKEN@github.com/account/repo.git where GIT_TOKEN is basically a placeholder for your token.
• At software environment creation time, we'll replace the placeholder with the personal access token.

@jose-moralez Let me know if something like this would work for you. Happy to discuss this further either here or in a call.

[jose-moralez]

Yeah I use a similar approach in the github action that runs some tests which need to clone one of the repos we use. Would there be like a secrets section in the coiled profile page? I think that would be extremely useful for some other credentials as well like for S3 and databases.

[marcosmoyano]

Would there be like a secrets section in the coiled profile page?

Exactly :+1:

[shughes-uk]

We have pretty good support for this now