openllb / hlb

A developer-first language to build and test any software efficiently
https://openllb.github.io/hlb/
Apache License 2.0
108 stars 12 forks source link

Add support for running modules from git scheme uri #312

Closed hinshun closed 2 years ago

hinshun commented 2 years ago

Supports new schemes for module uris:

On git:// and git+ssh:// mode, it relies on env $SSH_AUTH_SOCK, dials the agent and tests the SSH connection. If it fails on git+ssh://, then it fails hlb run. If it fails on git:// then it falls back to git+https://.

Known hosts from ~/.ssh/known_hosts are passed to llb.KnownSSHHosts(keys string).

❯ hlb run -t golang git://github.com/openllb/hlb:/go.hlb
[+] Building 4.2s (3/3) FINISHED
 => CACHED [module git://github.com/openllb/hlb:/go.hlb] git://github.com/openllb/hlb                                          3.0s
 => resolve image config for docker.io/library/golang:1.17.5-alpine                                                            1.0s
 => CACHED docker-image://docker.io/library/golang:1.17.5-alpine                                                               0.1s
 => => resolve docker.io/library/golang:1.17.5-alpine                                                                          0.1s

When running with either new scheme, it treats local relative to the ast.Directory the module is in. This means relative to the ref returned from gateway solve. This allows you to run remote hlb modules without cloning the repo.

Other commands also work with the new uri:

 ❯ hlb mod tree git://github.com/openllb/hlb
[+] Building 5.5s (2/2) FINISHED
 => resolve image config for docker.io/openllb/go.hlb:latest                                                                   4.9s
 => [import go] docker-image://docker.io/openllb/go.hlb:latest                                                                 0.5s
 => => resolve docker.io/openllb/go.hlb:latest                                                                                 0.2s
 => => sha256:f05b0316512e2b5fce823ed50c11990ab8448b9eef9d0644f490c0f77d5a782e 1.95kB / 1.95kB                                 0.3s
 => => extracting sha256:f05b0316512e2b5fce823ed50c11990ab8448b9eef9d0644f490c0f77d5a782e                                      0.0s
git@github.com/openllb/hlb/build.hlb
├── [go]  /go.hlb
│   └── [go]  sha256:910cae9/module.hlb
└── [mkdocs]  /mkdocs.hlb
    └── [go]  sha256:910cae9/module.hlb

Including debugging:

❯ hlb run --debug -t golang git://github.com/openllb/hlb:/go.hlb
#1 [module git://github.com/openllb/hlb:/go.hlb] git://github.com/openllb/hlb
#1 1.400 ref: refs/heads/master HEAD
#1 1.515 0bbdeac4d615b51d5991f92588dd291bfb376085   HEAD
#1 2.789 0bbdeac4d615b51d5991f92588dd291bfb376085   refs/heads/master
#1 CACHED
Type help for a list of commands
(hlb) s
git@github.com/openllb/hlb/go.hlb:13:4:
9  │ string versionCmd() {
10 │    "git describe --match 'v[0-9]*' --tags --dirty='.dirty' --always | sed 's/^v//'"
11 │ }
12 │
13 │ fs golang() {
   │    ^^^^^^
14 │    image "golang:1.17.5-alpine"
15 │ }
16 │
17 │ fs build(fs src, string package, string verPackage) {
(hlb)
hinshun commented 2 years ago

@aaronlehmann Also currently this PR doesn't utilize llb.KnownSSHHosts so its not secure atm.

Earthly for example, loads user's known hosts: https://github.com/earthly/earthly/blob/main/buildcontext/gitlookup.go#L589-L595

And has some hard coded keyscans for popular hosts (github.com, bitbucket.com): https://github.com/earthly/earthly/blob/main/buildcontext/gitlookup.go#L60-L62

hinshun commented 2 years ago

Updated description of PR.