rodjek / librarian-puppet

http://librarian-puppet.com
MIT License
693 stars 209 forks source link

Bug when using ":path" or ":git" in Puppetfile #272

Closed bvis closed 9 years ago

bvis commented 9 years ago

I found how to reproduce the bug. This is detected when you are using a Puppet forge mirror, because there you'll have your own modules and a mirror of the original Puppet forge.

How to reproduce

Create two puppet modules with a simple structure

metadata.json [tester/tester]

{
    "name": "tester-tester",
    "version": "0.1.0",
    "author": "Basilio Vera",
    "summary": "Just our own test",
    "license": "MIT",
    "dependencies": [
        { "name": "puppetlabs/inifile" },
        { "name": "tester/tester_dependency1" }
    ]
}

metadata.json [tester/tester_dependency1]

{
    "name": "tester-tester_dependency1",
    "version": "0.1.0",
    "author": "Basilio Vera",
    "summary": "Just our own inner dependency for the test",
    "license": "MIT",
    "dependencies": [
    ]
}

Upload them to your Forge, in this case named puppet-library.redtonic

Create a Puppetfile

forge 'http://puppet-library.redtonic'
mod 'tester/tester'

Execute the install

[vagrant@example puppet-test]$ librarian-puppet install --verbose
[Librarian] Ruby Version: 1.9.3
[Librarian] Ruby Platform: x86_64-linux
[Librarian] Rubygems Version: 2.4.3
[Librarian] Librarian Version: 0.1.2
[Librarian] Librarian Adapter: puppet
[Librarian] Librarian Adapter Version: 2.0.0
[Librarian] Project: /home/vagrant/puppet-test
[Librarian] Specfile: Puppetfile
[Librarian] Lockfile: Puppetfile.lock
[Librarian] Git: /usr/bin/git
[Librarian] Git Version: 1.7.11.3
[Librarian] Git Environment Variables:
[Librarian]   (empty)
[Librarian] Pre-Cached Sources:
[Librarian]   [:forge, "http://puppet-library.redtonic", {}]
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://puppet-library.redtonic", {}]
[Librarian] The specfile is unchanged: nothing to do.
[Librarian] Install: dependencies resolved
[Librarian] Pre-Cached Sources:
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://puppet-library.redtonic", {}]
[Librarian] Installing puppetlabs-inifile/1.2.0 <http://puppet-library.redtonic>
[Librarian] Executing puppet module install for puppetlabs-inifile 1.2.0: puppet module install --version 1.2.0 --target-dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/puppetlabs-inifile/1.2.0 --module_repository http://puppet-library.redtonic --modulepath /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/puppetlabs-inifile/1.2.0 --module_working_dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/puppetlabs-inifile/1.2.0 --ignore-dependencies puppetlabs-inifile
[Librarian] Installing tester-tester_dependency1/0.1.0 <http://puppet-library.redtonic>
[Librarian] Executing puppet module install for tester-tester_dependency1 0.1.0: puppet module install --version 0.1.0 --target-dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester_dependency1/0.1.0 --module_repository http://puppet-library.redtonic --modulepath /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester_dependency1/0.1.0 --module_working_dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester_dependency1/0.1.0 --ignore-dependencies tester-tester_dependency1
[Librarian] Installing tester-tester/0.1.0 <http://puppet-library.redtonic>
[Librarian] Executing puppet module install for tester-tester 0.1.0: puppet module install --version 0.1.0 --target-dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester/0.1.0 --module_repository http://puppet-library.redtonic --modulepath /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester/0.1.0 --module_working_dir /home/vagrant/puppet-test/.tmp/librarian/cache/source/puppet/forge/puppet-library_redtonic/tester-tester/0.1.0 --ignore-dependencies tester-tester

It works correctly!

Try the same with local/git paths

forge 'http://puppet-library.redtonic'
mod 'tester/tester',
  :path => '~/puppet-tester'

Re-install the packages using the local reference (the same happens when using ":git" references)

Install again with local reference with a clean installation.

[vagrant@example puppet-test]$ librarian-puppet install --verbose
[Librarian] Ruby Version: 1.9.3
[Librarian] Ruby Platform: x86_64-linux
[Librarian] Rubygems Version: 2.4.3
[Librarian] Librarian Version: 0.1.2
[Librarian] Librarian Adapter: puppet
[Librarian] Librarian Adapter Version: 2.0.0
[Librarian] Project: /home/vagrant/puppet-test
[Librarian] Specfile: Puppetfile
[Librarian] Lockfile: Puppetfile.lock
[Librarian] Git: /usr/bin/git
[Librarian] Git Version: 1.7.11.3
[Librarian] Git Environment Variables:
[Librarian]   (empty)
[Librarian] Pre-Cached Sources:
[Librarian]   [:forge, "http://puppet-library.redtonic", {}]
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://puppet-library.redtonic", {}]
[Librarian]   [:path, "~/puppet-tester", {}]
[Librarian] Analyzing spec and lock:
[Librarian]   Removed:
[Librarian]   ExplicitRemoved:
[Librarian]   Added:
[Librarian]   NonMatchingAdded:
[Librarian]   Changed:
[Librarian]     tester-tester
[Librarian]   DeepKeep:
[Librarian]   ShallowStrip:
[Librarian]     tester-tester
[Librarian] Resolving tester-tester (>= 0) <~/puppet-tester>
[Librarian]   Checking manifests
[Librarian]     Checking tester-tester/0.1.0 <~/puppet-tester>
[Librarian]       Resolving puppetlabs-inifile (>= 0) <https://forgeapi.puppetlabs.com>
[Librarian]         Checking manifests
[Librarian]   Module puppetlabs-inifile found versions: 1.2.0, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0, 1.0.4, 1.0.3, 1.0.1, 1.0.0
[Librarian]           Checking puppetlabs-inifile/1.2.0 <https://forgeapi.puppetlabs.com>
[Librarian]             Resolving tester-tester_dependency1 (>= 0) <https://forgeapi.puppetlabs.com>
[Librarian]               Checking manifests
Unable to find module 'tester-tester_dependency1' on https://forgeapi.puppetlabs.com
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/source/forge/repo_v3.rb:42:in `get_module'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/source/forge/repo_v3.rb:19:in `get_versions'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/source/forge/repo.rb:15:in `versions'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/source/forge/repo.rb:42:in `manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/source/forge.rb:144:in `manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/dependency.rb:117:in `cache_manifests!'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/dependency.rb:113:in `manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:152:in `block in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:165:in `block (2 levels) in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:179:in `block in scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:178:in `scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:164:in `block in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:163:in `scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:151:in `resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:62:in `recursive_resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:70:in `block in recursive_resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:154:in `block (3 levels) in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:187:in `block in scope_checking_manifest'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:186:in `scope_checking_manifest'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:153:in `block (2 levels) in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:211:in `block in map_find'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `each'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `map_find'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:152:in `block in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:165:in `block (2 levels) in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:179:in `block in scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:178:in `scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:164:in `block in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:163:in `scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:151:in `resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:62:in `recursive_resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:70:in `block in recursive_resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:154:in `block (3 levels) in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:187:in `block in scope_checking_manifest'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:186:in `scope_checking_manifest'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:153:in `block (2 levels) in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:211:in `block in map_find'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `each'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:210:in `map_find'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:152:in `block in resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:165:in `block (2 levels) in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:179:in `block in scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:178:in `scope_checking_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:164:in `block in scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:163:in `scope_resolving_dependency'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:151:in `resolving_dependency_map_find_manifests'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:62:in `recursive_resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:50:in `resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/resolver.rb:23:in `resolve'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/action/resolve.rb:26:in `run'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:169:in `resolve!'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/lib/librarian/puppet/cli.rb:67:in `install'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block (2 levels) in bin!'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:31:in `returning_status'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block in bin!'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:47:in `with_environment'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `bin!'
/usr/local/rvm/gems/ruby-1.9.3-p550/gems/librarian-puppet-2.0.0/bin/librarian-puppet:7:in `<top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p550/bin/librarian-puppet:23:in `load'
/usr/local/rvm/gems/ruby-1.9.3-p550/bin/librarian-puppet:23:in `<main>'
/usr/local/rvm/gems/ruby-1.9.3-p550/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-1.9.3-p550/bin/ruby_executable_hooks:15:in `<main>'
carlossg commented 9 years ago

thanks for the detailed report, can you create a test case in https://github.com/rodjek/librarian-puppet/blob/master/features/install/forge.feature? it should be easy with the examples already there

bvis commented 9 years ago

I can add these test cases:

@other-forge
  Scenario: Installing a module from another forge
    Given a file named "Puppetfile" with:
    """
    forge "http://172.20.0.238"

    mod 'tester/tester'
    """
    When I run `librarian-puppet install`
    Then the exit status should be 0
    And the file "modules/tester/metadata.json" should match /"name": "tester-tester"/
    And the file "modules/tester_dependency1/metadata.json" should match /"name": "tester-tester_dependency1"/

@other-forge
Scenario: Installing a module from another forge with local reference
    Given a file named "Puppetfile" with:
    """
    forge "http://172.20.0.238"

    mod 'tester/tester',
      :path => './tester-tester'
    """
    And a file named "tester-tester/metadata.json" with:
    """
    {
        "name": "tester-tester",
        "version": "0.1.0",
        "author": "Basilio Vera",
        "summary": "Just our own test",
        "license": "MIT",
        "dependencies": [
            { "name": "puppetlabs/inifile" },
            { "name": "tester/tester_dependency1" }
        ]
    }
    """

    When I run `librarian-puppet install`
    Then the exit status should be 0
    And the file "modules/tester/metadata.json" should match /"name": "tester-tester"/
    And the file "modules/tester_dependency1/metadata.json" should match /"name": "tester-tester_dependency1"/

I already have tried them with the expected result:

$ cucumber --tag @other-forge
Feature: cli/install/forge
  Puppet librarian needs to install modules from the Puppet Forge

  @other-forge
  Scenario: Installing a module from another forge                                                             # features/install/forge.feature:254
    Given a file named "Puppetfile" with:                                                                      # aruba-0.6.1/lib/aruba/cucumber.rb:29
      """
      forge "http://172.20.0.238"

      mod 'tester/tester'
      """
    When I run `librarian-puppet install`                                                                      # aruba-0.6.1/lib/aruba/cucumber.rb:93
    Then the exit status should be 0                                                                           # aruba-0.6.1/lib/aruba/cucumber.rb:197
    And the file "modules/tester/metadata.json" should match /"name": "tester-tester"/                         # aruba-0.6.1/lib/aruba/cucumber.rb:359
    And the file "modules/tester_dependency1/metadata.json" should match /"name": "tester-tester_dependency1"/ # aruba-0.6.1/lib/aruba/cucumber.rb:359

  @other-forge
  Scenario: Installing a module from another forge with local reference                                        # features/install/forge.feature:267
    Given a file named "Puppetfile" with:                                                                      # aruba-0.6.1/lib/aruba/cucumber.rb:29
      """
      forge "http://172.20.0.238"

      mod 'tester/tester',
        :path => './tester-tester'
      """
    And a file named "tester-tester/metadata.json" with:                                                       # aruba-0.6.1/lib/aruba/cucumber.rb:29
      """
      {
          "name": "tester-tester",
          "version": "0.1.0",
          "author": "Basilio Vera",
          "summary": "Just our own test",
          "license": "MIT",
          "dependencies": [
              { "name": "puppetlabs/inifile" },
              { "name": "tester/tester_dependency1" }
          ]
      }
      """
    When I run `librarian-puppet install`                                                                      # aruba-0.6.1/lib/aruba/cucumber.rb:93
    Then the exit status should be 0                                                                           # aruba-0.6.1/lib/aruba/cucumber.rb:197
      Exit status was 1 but expected it to be 0. Output:

      Unable to find module 'tester-tester_dependency1' on https://forgeapi.puppetlabs.com

       (RSpec::Expectations::ExpectationNotMetError)
      features/install/forge.feature:291:in `Then the exit status should be 0'
    And the file "modules/tester/metadata.json" should match /"name": "tester-tester"/                         # aruba-0.6.1/lib/aruba/cucumber.rb:359
    And the file "modules/tester_dependency1/metadata.json" should match /"name": "tester-tester_dependency1"/ # aruba-0.6.1/lib/aruba/cucumber.rb:359

Failing Scenarios:
cucumber features/install/forge.feature:267 # Scenario: Installing a module from another forge with local reference

2 scenarios (1 failed, 1 passed)
11 steps (1 failed, 2 skipped, 8 passed)
0m30.776s

But this means that you need a forge proxy configured at this IP with this modules inside. I did it following these easy steps:

mkdir modules
gem install puppet-forge-server
puppet-forge-server -m modules/ -x https://forgeapi.puppetlabs.com --port 80 --bind-host 0.0.0.0

But this makes these tests to not to be portable, at least if you do not create this server under a public and widely accessible IP. How would you like to proceed?

Alternative

One possible option would be to remove the first test and check that the error obtained in the second test is related with the non existing "local" forge instead of the _Unable to find module 'tester-testerdependency1' on https://forgeapi.puppetlabs.com

carlossg commented 9 years ago

Could try just checking the librarian-puppet install --verbose output, it shouldn't include forgeapi.puppetlabs.com even if the command fails. There are testing modules already in git

bvis commented 9 years ago

As you may see I've created a pull request with your suggestion.