Closed kevindrosendahl closed 6 years ago
seem to have basic references implemented, but havent integrated it or added template parsing to it
re-integrating the template engine into the resolution.
hoping to ship this in the next day or two.
semver/branch resolution working, but now re-doing the template engine including handling secret passing
finishing up adding a caching layer, then hopefully should have all of the pieces
k8s api caching layer almost done. added two new custom resources, Template
and GitTemplate
. Template
contains the non-hydrated version of the template (parameters not substituted) and GitTemplate
stores a pointer from a tuple of (git url, commit, file path)
to a Template
.
Also was in the area so added YAML support.
Took a side quest last night to investigate how much we can reduce the amount of data we're fetching from the git repo when trying to just get a single file, but Github has disabled pretty much every avenue of doing this. I think this will actually be a pretty pressing problem now that we're trying to encourage people to have their lattice definitions in their app repos, don't want to have to clone every single app into the api server. Popping off this for now, but looks like we may want to use Github's API.
Was thinking about the latter part of this in the meeting this morning - what if you do something like
git clone git@github.com:mlab-lattice/system__petflix.git --no-checkout
cd system_petflix
git update-index --skip-worktree $(git ls-files)
git update-index --no-skip-worktree (cherrypick anything you want)
git checkout -- *
git pull
That'll leave you a repo pulled with just meta info about itself and whatever files you specifically added.
when you clone there you've still downloaded the whole git database:
[Tue 24 12:51:01] kevinrosendahl at kevin-mlab.local in ~/t/git
[0] time git clone git@github.com:mlab-lattice/lattice.git -n
Cloning into 'lattice'...
remote: Counting objects: 23544, done.
remote: Compressing objects: 100% (219/219), done.
remote: Total 23544 (delta 100), reused 239 (delta 66), pack-reused 23239
Receiving objects: 100% (23544/23544), 173.13 MiB | 6.58 MiB/s, done.
Resolving deltas: 100% (14608/14608), done.
git clone git@github.com:mlab-lattice/lattice.git -n 12.16s user 3.24s system 48% cpu 32.071 total
[Tue 24 12:51:37] kevinrosendahl at kevin-mlab.local in ~/t/git
[0] du -sh lattice
177M lattice
the --no-checkout just doesn't make the working director the copy of the worktree from HEAD
for posterity, my attempts led me basically to:
git init
git remote add $REPO_URL
git fetch --depth 1 $COMMIT_SHA
git archive --remote=$REPO_URL ${COMMIT_SHA}:path/to/directory filename
git-archive
https://www.gilesorr.com/blog/git-archive-github.htmlcurl -L $REPO_URL/archive/$COMMIT_SHA.tar.gz
🤦♂️ shoot i should've tried that with a larger repo...
still chipping away. as i got deeper i realized how much of the tree based deploy stuff depends on the implementation details here, so it's taking a longer than i hoped but hopefully setting up for future success
seems to mostly be working. need to support not setting a git_repository
section of a service's build
in order to implicitly build the repo the definition is in. other than that tightening up some tests and hopefully adding in some light docs and shouldbe good to go
@alek-mlab should try this and report back
should be able to say something like: