search

chef / cookbook-omnifetch

Fetch Chef Cookbooks from Various Sources to a Local Cache
Apache License 2.0
6 stars 9 forks source link

Fetching repo with git-lfs files fails #21

Open Aketzu opened 6 years ago

Aketzu commented 6 years ago

Description

When I try to run chef install Policyfile.rb the run will end with Smudge error: Error downloading from git-lfs.

Not sure whether the problem is in the way cookbook-omnifetch uses git or in git-lfs not providing support for this case.

ChefDK Version

Chef Development Kit Version: 2.3.4 chef-client version: 13.4.19 delivery version: master (73ebb72a6c42b3d2ff5370c476be800fee7e5427) berks version: 6.3.1 kitchen version: 1.17.0 inspec version: 1.36.1

Platform Version

Debian 9.1 (docker bitnami/minideb:latest)

Log

Log of the run in https://gist.github.com/Aketzu/79bafc57d0c276acfe06acfc9b9bf31c

Main points:

root@d425dc502635:~# GIT_CURL_VERBOSE=1 GIT_TRACE=1 chef install Policyfile.rb -D                                            
...
Installing payara  >= 0.0.0 from git
CMD: git clone git@git.vilant.com:chef/payara-cookbook.git "/root/.chefdk/cache/.cache/git/3b3a7b287f96b4a8f60a36fe4b89cece06c17e5a" --bare --no-hardlinks
CMD: git rev-parse master
CMD: git clone --no-checkout "/root/.chefdk/cache/.cache/git/3b3a7b287f96b4a8f60a36fe4b89cece06c17e5a" "/tmp/d20171116-1143-8dudo"
CMD: git fetch --force --tags "/root/.chefdk/cache/.cache/git/3b3a7b287f96b4a8f60a36fe4b89cece06c17e5a"
CMD: git reset --hard c58c214e0584e750a06a1601d83dd17364c849f9
Error: Failed to generate Policyfile.lock
Reason: (CookbookOmnifetch::GitCommandError) Git error: command `git reset --hard c58c214e0584e750a06a1601d83dd17364c849f9` failed. If this error persists, try removing the cache directory at '/root/.chefdk/cache/.cache/git/3b3a7b287f96b4a8f60a36fe4b89cece06c17e5a'.Output from the command:

13:32:10.187557 git.c:371               trace: built-in: git 'reset' '--hard' 'c58c214e0584e750a06a1601d83dd17364c849f9'
13:32:10.189405 run-command.c:350       trace: run_command: 'git-lfs filter-process'
13:32:10.189673 run-command.c:209       trace: exec: '/bin/sh' '-c' 'git-lfs filter-process' 'git-lfs filter-process'
13:32:10.198635 trace git-lfs: run_command: 'git' version
13:32:10.201844 trace git-lfs: run_command: 'git' config -l
13:32:10.203748 trace git-lfs: Install hook: pre-push, force=false, path=/tmp/d20171116-1143-8dudo/.git/hooks/pre-push
13:32:10.204060 trace git-lfs: Install hook: post-checkout, force=false, path=/tmp/d20171116-1143-8dudo/.git/hooks/post-checkout
13:32:10.204264 trace git-lfs: Install hook: post-commit, force=false, path=/tmp/d20171116-1143-8dudo/.git/hooks/post-commit
13:32:10.204487 trace git-lfs: Install hook: post-merge, force=false, path=/tmp/d20171116-1143-8dudo/.git/hooks/post-merge
13:32:10.204655 trace git-lfs: Initialize filter-process
13:32:10.204740 trace git-lfs: Read filter-process request.
Downloading files/assembly-8.1.0.zip (29 MB)
13:32:10.204952 trace git-lfs: tq: running as batched queue, batch size of 100
13:32:10.205025 trace git-lfs: tq: sending batch of size 1
Error downloading object: files/assembly-8.1.0.zip (70bdc41): Smudge error: Error downloading files/assembly-8.1.0.zip (70bdc4131f0d4be92660574a193053b7c008ade2af0c5cd5a0de74c30b5c0cc6): batch request: missing protocol: "/root/.chefdk/cache/.cache/git/3b3a7b287f96b4a8f60a36fe4b89cece06c17e5a.git/info/lfs"

Errors logged to /tmp/d20171116-1143-8dudo/.git/lfs/objects/logs/20171116T133210.205150881.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: files/assembly-8.1.0.zip: smudge filter lfs failed

/opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/cookbook-omnifetch-0.6.0/lib/cookbook-omnifetch/git.rb:174:in `git'
/opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/cookbook-omnifetch-0.6.0/lib/cookbook-omnifetch/git.rb:60:in `block in install'
/opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/cookbook-omnifetch-0.6.0/lib/cookbook-omnifetch/git.rb:58:in `chdir'
/opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/cookbook-omnifetch-0.6.0/lib/cookbook-omnifetch/git.rb:58:in `install'

Thoughts

Since cookbook-omnifetch is doing clone --bare + clone + reset the first repo will not contain git-lfs files and subsequent reset will fail to connect the dots between clone and real original repo (GitLab server in this case).

If I do just git clone (without --bare) I get the full repo with lfs files so permissions and URLs work in the normal case.

In this case command is run in a clean docker container every time so caching the repo is pointless but there is no option to disable caching.

Aketzu commented 6 years ago

As noted in the refereced issue this is unsupported way from git-lfs side. So it might be easier to change cookbook-omnifetch in the short term.

Background about the case: We use policyfile to distribute cookbooks to a server behind firewalls and transferring a single package is the easiest way. Also due to firewalls we need to include all related software (Java, Payara, apps) in the cookbooks and storing large binary blobs in Git repo is rather inefficient without lfs support.

tyler-ball commented 5 years ago

We need to check if this is still an issue