search

drrb / puppet-library

A private Puppet Forge
GNU General Public License v3.0
87 stars 20 forks source link

Proxying Puppet Forge not working as expected #1

Closed josepedrocorreia closed 10 years ago

josepedrocorreia commented 10 years ago

Hi,

I'm using this in conjunction with librarian-puppet. My aim is to have a server that both proxies the Puppet Forge and serves some additional modules from a directory.

I'm running into a problem though. librarian-puppet seems to be able to retrieve meta information from the server (like dependencies and such) but there seems to be a problem when the Puppet command to install the module is fired. Here's the log from librarian-puppet:

[Librarian] Ruby Version: 1.8.7
[Librarian] Ruby Platform: x86_64-linux
[Librarian] Rubygems Version: 1.3.7
[Librarian] Librarian Version: 0.0.24
[Librarian] Librarian Adapter: puppet
[Librarian] Project: /etc/puppet
[Librarian] Specfile: Puppetfile
[Librarian] Lockfile: Puppetfile.lock
[Librarian] Git: /usr/bin/git
[Librarian] Git Version: git version 1.7.1
[Librarian] Git Environment Variables:
[Librarian]   (empty)
[Librarian] Pre-Cached Sources:
[Librarian]   [:forge, "http://tk132:9292", {}]
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://tk132:9292", {}]
[Librarian] The specfile is unchanged: nothing to do.
[Librarian] Pre-Cached Sources:
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://tk132:9292", {}]
Error: Could not download module: 404 Not Found 
Error: Try 'puppet help module install' for usage
Error executing puppet module install:
puppet module install --target-dir '/etc/puppet/.tmp/librarian/cache/source/puppet/forge/befcb58a528b8117d41eb8a44f277876/domcleal/augeasproviders/version/b24db5c8391ec5c5d93cd0ac083acbdf' --module_repository 'http://tk132:9292' --modulepath '/etc/puppet/.tmp/librarian/cache/source/puppet/forge/befcb58a528b8117d41eb8a44f277876/domcleal/augeasproviders/version/b24db5c8391ec5c5d93cd0ac083acbdf' --ignore-dependencies 'domcleal/augeasproviders'
Error:
Notice: Preparing to install into /etc/puppet/.tmp/librarian/cache/source/puppet/forge/befcb58a528b8117d41eb8a44f277876/domcleal/augeasproviders/version/b24db5c8391ec5c5d93cd0ac083acbdf ...
Notice: Downloading from http://tk132:9292 ...

/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/lib/librarian/puppet/source/forge.rb:105:in `cache_version_unpacked!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/lib/librarian/puppet/source/forge.rb:48:in `install_version!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/lib/librarian/puppet/source/forge.rb:236:in `install!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/manifest.rb:191:in `install!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/action/install.rb:49:in `install_manifests'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/action/install.rb:48:in `each'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/action/install.rb:48:in `install_manifests'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/action/install.rb:39:in `perform_installation'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/action/install.rb:12:in `run'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/cli.rb:157:in `install!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/lib/librarian/puppet/cli.rb:64:in `install'
/usr/lib/ruby/gems/1.8/gems/thor-0.18.1/lib/thor/command.rb:27:in `__send__'
/usr/lib/ruby/gems/1.8/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
/usr/lib/ruby/gems/1.8/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
/usr/lib/ruby/gems/1.8/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
/usr/lib/ruby/gems/1.8/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/vendor/librarian/lib/librarian/cli.rb:29:in `bin!'
/usr/lib/ruby/gems/1.8/gems/librarian-puppet-0.9.10/bin/librarian-puppet:9
/usr/bin/librarian-puppet:19:in `load'
/usr/bin/librarian-puppet:19

It looks to me like the original problem is that librarian-puppet cannot download the archive. This seems to be confirmed by the log from puppet-library:

# puppet-library --proxy http://forge.puppetlabs.com --port 9292
/usr/lib/ruby/gems/1.8/gems/haml-4.0.5/lib/haml/parser.rb:86: warning: regexp has invalid interval
Starting Puppet Library server:
 |- Port: 9292
 |- Host: default
 |- Server: default
 `- Forges:
    - PuppetLibrary::Forge::Proxy: http://forge.puppetlabs.com
[2014-02-05 18:12:14] INFO  WEBrick 1.3.1
[2014-02-05 18:12:14] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
[2014-02-05 18:12:14] WARN  TCPServer Error: Address already in use - bind(2)
[2014-02-05 18:12:14] INFO  WEBrick::HTTPServer#start: pid=26959 port=9292
192.168.0.132 - - [05/Feb/2014 18:12:25] "GET /api/v1/releases.json?module=domcleal/augeasproviders HTTP/1.1" 200 2596 0.6781
tk132 - - [05/Feb/2014:18:12:24 CET] "GET /api/v1/releases.json?module=domcleal/augeasproviders HTTP/1.1" 200 2596
- -> /api/v1/releases.json?module=domcleal/augeasproviders
192.168.0.132 - - [05/Feb/2014 18:12:30] "GET /system/releases/d/domcleal/domcleal-augeasproviders-1.0.2.tar.gz HTTP/1.1" 404 624 0.0006
tk132 - - [05/Feb/2014:18:12:30 CET] "GET /system/releases/d/domcleal/domcleal-augeasproviders-1.0.2.tar.gz HTTP/1.1" 404 624
- -> /system/releases/d/domcleal/domcleal-augeasproviders-1.0.2.tar.gz

My Puppetfile is simply:

forge 'http://tk132:9292'
mod 'domcleal/augeasproviders'

Any clues what might be going wrong?

drrb commented 10 years ago

Thanks for the simple example and detailed output.

Looks like this was introduced in Version 0.3.0, where queries to proxies started being passed more directly through to the proxied servers (to improve performance and internal APIs). Unfortunately, the download paths from the the proxied server (/system/releases/d/domcleal/domcleal-augeasproviders-1.0.2.tar.gz in your example) were being passed back to the client, which resulted in puppet-library attempting to download modules using the wrong paths on the proxy (it should have been /modules/domcleal-augeasproviders-1.0.2.tar.gz in your example).

A fix just went in to 438399c0054ca731e136fa64b1b8401d16d2bbb7, and I just released Version 0.7.0, which contains the fix.

Feel free to reopen if you still find the issue with 0.7.0.

josepedrocorreia commented 10 years ago

This seems to work, thanks.