add references to system definition

[kevindrosendahl]

should be able to say something like:

{
    "type": "v1/system",

    "components": {
        "foo": {
            "type": "v1/reference",

            "git_repository": {
                 "url": "https://github.com/foo/bar",
                 "tag": "~1.0.0"
            }
        }
    }
}

[kevindrosendahl]

seem to have basic references implemented, but havent integrated it or added template parsing to it

[kevindrosendahl]

re-integrating the template engine into the resolution.

hoping to ship this in the next day or two.

[kevindrosendahl]

semver/branch resolution working, but now re-doing the template engine including handling secret passing

[kevindrosendahl]

finishing up adding a caching layer, then hopefully should have all of the pieces

[kevindrosendahl]

k8s api caching layer almost done. added two new custom resources, Template and GitTemplate. Template contains the non-hydrated version of the template (parameters not substituted) and GitTemplate stores a pointer from a tuple of (git url, commit, file path) to a Template.

Also was in the area so added YAML support.

Took a side quest last night to investigate how much we can reduce the amount of data we're fetching from the git repo when trying to just get a single file, but Github has disabled pretty much every avenue of doing this. I think this will actually be a pretty pressing problem now that we're trying to encourage people to have their lattice definitions in their app repos, don't want to have to clone every single app into the api server. Popping off this for now, but looks like we may want to use Github's API.

[navenoxin]

Was thinking about the latter part of this in the meeting this morning - what if you do something like

git clone git@github.com:mlab-lattice/system__petflix.git --no-checkout
cd system_petflix
git update-index --skip-worktree $(git ls-files)
git update-index --no-skip-worktree (cherrypick anything you want)
git checkout -- *
git pull

That'll leave you a repo pulled with just meta info about itself and whatever files you specifically added.

[kevindrosendahl]

when you clone there you've still downloaded the whole git database:

[Tue 24 12:51:01] kevinrosendahl at kevin-mlab.local in ~/t/git
[0] time git clone git@github.com:mlab-lattice/lattice.git -n
Cloning into 'lattice'...
remote: Counting objects: 23544, done.
remote: Compressing objects: 100% (219/219), done.
remote: Total 23544 (delta 100), reused 239 (delta 66), pack-reused 23239
Receiving objects: 100% (23544/23544), 173.13 MiB | 6.58 MiB/s, done.
Resolving deltas: 100% (14608/14608), done.
git clone git@github.com:mlab-lattice/lattice.git -n  12.16s user 3.24s system 48% cpu 32.071 total
[Tue 24 12:51:37] kevinrosendahl at kevin-mlab.local in ~/t/git
[0] du -sh lattice
177M    lattice

the --no-checkout just doesn't make the working director the copy of the worktree from HEAD

[kevindrosendahl]

for posterity, my attempts led me basically to:

• https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository
  • git init
  • git remote add $REPO_URL
  • git fetch --depth 1 $COMMIT_SHA
  • however, Github does not seem to have configured uploadpack.allowAnySHA1InWant but instead only uploadpackallow.ReachableSHA1InWant which means that you can only fetch commit SHAs that are directly pointed at by refs (branches/tags), so not just any commit object or worktree or blob.
• https://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository-sparse-checkout
  • git archive --remote=$REPO_URL ${COMMIT_SHA}:path/to/directory filename
  • Github doesn't support git-archive https://www.gilesorr.com/blog/git-archive-github.html
  • can use curl -L $REPO_URL/archive/$COMMIT_SHA.tar.gz
  • this'll download the whole worktree of that commit, but that's still much better
  • need OAUTH token for private repo though
  • at that point might as well use github blob API (https://developer.github.com/v3/git/blobs/)

[navenoxin]

🤦‍♂️ shoot i should've tried that with a larger repo...

[kevindrosendahl]

still chipping away. as i got deeper i realized how much of the tree based deploy stuff depends on the implementation details here, so it's taking a longer than i hoped but hopefully setting up for future success

[kevindrosendahl]

seems to mostly be working. need to support not setting a git_repository section of a service's build in order to implicitly build the repo the definition is in. other than that tightening up some tests and hopefully adding in some light docs and shouldbe good to go

[doug-ol]

@alek-mlab should try this and report back

[kevindrosendahl]

add in https://github.com/mlab-lattice/lattice/pull/186