search

github / octocatalog-diff

Compile Puppet catalogs from 2 branches, versions, etc., and compare them
https://rubygems.org/gems/octocatalog-diff/
MIT License
239 stars 85 forks source link

Issue with git archive on Ubuntu 16.04 #133

Open petems opened 7 years ago

petems commented 7 years ago

Description of problem

Test: role::vanilla on CentOS-7.0-64 Exit: 1 Status: failed Errors: E, [2017-06-17T18:58:01.369349 #4056] ERROR -- : Git checkout error: Git archive add_jenkinsfile->/tmp/d20170617-4056-13m9i4l failed: STDOUT: STDERR: fatal: Not a valid object name tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors E, [2017-06-17T18:58:01.369968 #4053] ERROR -- : Git checkout error: Git archive production->/tmp/d20170617-4053-qxyqqv failed: STDOUT: STDERR: fatal: Not a valid object name tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors OctocatalogDiff::Errors::BootstrapError: Git archive add_jenkinsfile->/tmp/d20170617-4056-13m9i4l failed: STDOUT: STDERR: fatal: Not a valid object name tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:109:in rescue in git_checkout' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:104:ingit_checkout' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:93:in bootstrap_directory' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:130:inbootstrap' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:141:in build_catalog' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:64:inbuild' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog.rb:59:in build' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:237:inbuild_catalog' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:in call' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:inexecute' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:202:in execute_task' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:117:inblock (2 levels) in run_tasks_parallel' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:in fork' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:inblock in run_tasks_parallel' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in each' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:ineach_with_index' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in run_tasks_parallel' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:93:inrun_tasks' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:92:in build_catalog_parallelizer' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:29:incatalogs' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1/catalog-diff.rb:34:in catalog_diff' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1.rb:19:incatalog_diff' /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/cli.rb:112:in cli' /usr/local/bundle/gems/octocatalog-diff-1.3.0/bin/octocatalog-diff:34:in<top (required)>' /usr/local/bundle/bin/octocatalog-diff:17:in load' /usr/local/bundle/bin/octocatalog-diff:17:in<top (required)>'


- What did you expect to happen?
Comparison between to branches to work.

- How can someone reproduce the problem?
For context, I'm using a custom plugin that runs an r10k run of a Puppetfile in a temporary folder, then runs an octocatalog-diff between the two runs.

- Command used and debugging output
The actual octocatalog-diff command is this:

> bundle exec octocatalog-diff --fact-file /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v/spec/factsets/Ubuntu-16.04-64.yaml --from production --to add_jenkinsfile --basedir /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v --puppet-binary /var/lib/jenkins/workspace/ontrol-repo_add_jenkinsfile-YF67IPOJSZEA4N7KRTNFGNR644LHR2V3YPYE6BSAEQYMBAKBXGLQ/vendor/bundle/ruby/2.3.0/bin/puppet --bootstrap-script '/tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v/bootstrap_script.rb' --hiera-config /var/lib/jenkins/workspace/ontrol-repo_add_jenkinsfile-YF67IPOJSZEA4N7KRTNFGNR644LHR2V3YPYE6BSAEQYMBAKBXGLQ/spec/hiera.yaml --pass-env-vars BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT,RUBYLIB

> Platform and version information

- Your OS: `Ubuntu 16.04`
- Your Ruby version: `ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]`
- Your version of Puppet:

bundle exec puppet --version 4.7.1

- Your version of octocatalog-diff:

bundle exec octocatalog-diff --version octocatalog-diff 1.2.0


> Do the tests pass from a clean checkout?
Yes

> Anything else to add that you think will be helpful?
The same command works fine on OSX, but on Ubuntu I get the git archive error

Working on OSX shown below:

bundle exec onceover run diff --from production --to add_jenkinsfile --debug --trace DEBUG -> Reading /Users/petersouter/projects/psouter-control-repo/environment.conf DEBUG -> Searching puppet code for roles and profiles DEBUG -> Searching puppet code for roles and profiles DEBUG -> Reading factsets DEBUG -> Reading factsets DEBUG -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-1opdrc5 DEBUG -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-1gjpxqa DEBUG -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-vk4vtf DEBUG -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-tsy6ty INFO -> Preparing environment for role::vanilla on CentOS-7.0-64 INFO -> Preparing environment for role::vanilla on Ubuntu-16.04-64 DEBUG -> Creating temp directory DEBUG -> Temp directory created at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo DEBUG -> Copying controlrepo to /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo DEBUG -> Creating temp directory DEBUG -> Temp directory created at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1 DEBUG -> Copying controlrepo to /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1 DEBUG -> Deploying vendored factsets DEBUG -> Deploying vendored factsets INFO -> Deploying Puppetfile for role::vanilla on Ubuntu-16.04-64 INFO -> Deploying Puppetfile for role::vanilla on CentOS-7.0-64 DEBUG -> Creating before script that overwrites site.pp DEBUG -> Getting Puppet binary DEBUG -> Running Octocatalog diff INFO -> Compiling catalogs for role::vanilla on Ubuntu-16.04-64 DEBUG -> Running: bundle exec octocatalog-diff --fact-file /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1/spec/factsets/Ubuntu-16.04-64.yaml --from production --to add_jenkinsfile --basedir /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1 --puppet-binary /Users/petersouter/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/bin/puppet --bootstrap-script '/var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1/bootstrap_script.rb' --hiera-config /Users/petersouter/projects/psouter-control-repo/spec/hiera.yaml --pass-env-vars RUBYMOTION_ANDROID_SDK,RUBYMOTION_ANDROID_NDK,RUBYLIB,BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT DEBUG -> Creating before script that overwrites site.pp DEBUG -> Getting Puppet binary DEBUG -> Running Octocatalog diff INFO -> Compiling catalogs for role::vanilla on CentOS-7.0-64 DEBUG -> Running: bundle exec octocatalog-diff --fact-file /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo/spec/factsets/CentOS-7.0-64.yaml --from production --to add_jenkinsfile --basedir /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo --puppet-binary /Users/petersouter/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/bin/puppet --bootstrap-script '/var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo/bootstrap_script.rb' --hiera-config /Users/petersouter/projects/psouter-control-repo/spec/hiera.yaml --pass-env-vars RUBYMOTION_ANDROID_SDK,RUBYMOTION_ANDROID_NDK,RUBYLIB,BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT INFO -> Storing results for role::vanilla on CentOS-7.0-64 DEBUG -> Backing up modules to thread cache /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_CentOS-7.0-6420170618-24795-up8vmo DEBUG -> Removing temporary build cache INFO -> Storing results for role::vanilla on Ubuntu-16.04-64 DEBUG -> Backing up modules to thread cache /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/rolevanilla_on_Ubuntu-16.04-6420170618-24795-huqie1 DEBUG -> Removing temporary build cache Test: role::vanilla on CentOS-7.0-64 Exit: 0 Status: no differences

Test: role::vanilla on Ubuntu-16.04-64 Exit: 0 Status: no differences



I know that the plugin isn't developed by you folk, but this error is really weird looking, and it's strange that the exact same commands work on OSX but no Ubuntu, so I was hoping you'd be able to help. I can give any debug information needed 😄
kpaulisse commented 7 years ago

fatal: Not a valid object name is an error message coming from git. That's the root of the problem here (that output is piped to tar, so the complaints from tar are downstream from whatever's wrong).

The code actually being run here is: https://github.com/github/octocatalog-diff/blob/master/scripts/git-extract/git-extract.sh

My suggestion: cd to the directory where this code actually lives, and run the git archive command that octocatalog-diff is actually running. From your example:

cd /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v
git archive --format=tar add_jenkinsfile

Perhaps that will lead you to discover an issue, either with the repository itself, or with different command line arguments that are needed for the particular version of the git command needed for this platform. If there's something that needs to be fixed in git-extract.sh to make that work then I certainly welcome pull requests or suggestions on what you had to do to make the above command work in your environment.

asenci commented 4 years ago

FWIW, I had the same error using octocatalog-diff with GitHub Actions, turns out the "checkout" action only fetches the committed ref by default. Had to update my workflow to fetch both the source and target branches:

    steps:
      - name: Checkout target branch
        uses: actions/checkout@v2
        with:
          ref: ${{ github.base_ref }}
      - name: Checkout source branch
        run: |
          git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin ${{ github.head_ref }}
          git checkout --progress --force -B ${{ github.head_ref }} refs/remotes/origin/${{ github.head_ref }}