search

kindredgroup / puppet-forge-server

Private Puppet forge server supports local files and both v1 and v3 API proxies
69 stars 44 forks source link

Passenger support #1

Closed jskarpe closed 9 years ago

jskarpe commented 9 years ago

Want to run this behind Apache

i11 commented 9 years ago

roger that. I might have some time to look at it next week.

i11 commented 9 years ago

Here is an config.ru example for the current version:

require 'rubygems'
require 'puppet_forge_server'

# Set base cache directory for proxy backends 
cache_dir = '/opt/forge/cache' # default: File.join(Dir.tmpdir.to_s, 'puppet-forge-server', 'cache')

# Create backends
backends = [
  PuppetForgeServer::Backends::Directory.new('/opt/forge/modules'),
  PuppetForgeServer::Backends::ProxyV3.new('https://forgeapi.puppetlabs.com', cache_dir)
]

# Disable access logging, log errors to STDERR
PuppetForgeServer::Logger.set({:server => STDERR, :access => File.open(File::NULL, "w")})

# Run
run PuppetForgeServer::Server.new.build backends

Apache virtualhost config:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot /opt/forge/public
    <Directory /opt/forge/public>
        Allow from all
        Options -MultiViews
        Require all granted
    </Directory>
</VirtualHost>

Not too pretty, but it works.

jskarpe commented 9 years ago

You have a small typo: Directroy should be Directory

uninitialized constant PuppetForgeServer::Backends::Directroy (NameError)
  config.ru:7:in `block in <main>'
  /usr/lib/ruby/vendor_ruby/rack/builder.rb:55:in `instance_eval'
  /usr/lib/ruby/vendor_ruby/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'

Also;

When I'm browsing it, all I see is:

Not Found

No errors in log though?

Librarian is reporting the same thing (not found). Both for local modules in /opt/forge/modules and puppetlabs forge modules.

i11 commented 9 years ago

Yeah, I had a typo at first, but then the comment was updated. Please try again copying the current text.

jskarpe commented 9 years ago

Your updated config results in:

can't convert IO into Integer (TypeError)
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:27:in `initialize'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:27:in `new'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:27:in `initialize'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:60:in `new'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:60:in `block in set'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:59:in `each'
  /var/lib/gems/1.9.1/gems/puppet-forge-server-1.1.0/lib/puppet_forge_server/logger.rb:59:in `set'
  config.ru:14:in `block in <main>'
  /usr/lib/ruby/vendor_ruby/rack/builder.rb:55:in `instance_eval'
  /usr/lib/ruby/vendor_ruby/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'

Adding your previous logging option starts the server, but the Not found stuff is still present.

i11 commented 9 years ago

The last changes had to be released... Please update the gem to 1.2.0 and try again.

i11 commented 9 years ago

By the way there is nothing answering for the root context, so it's normal to get Not found for something like curl http://localhost. But actually the forge is ready to go and something like curl http://localhost/v3/modules/puppetlabs-stdlib or puppet module install puppetlabs-stdlib --module_repository http://localhost would work fine

jskarpe commented 9 years ago

Accessing the /v3/modules/puppetlabs-stdlib uri works fine, but Librarian doesn't seem too happy with it (404). Not sure what uri exactly it tries though.

# Puppetfile
forge "http://localhost"

Also, I'm unable to find my module uploaded to /opt/forge/modules

What's a working example of this?

i11 commented 9 years ago

Your Puppetfile looks fine to me. Here is the one I used to test it:

cat > Puppetfile <<EOF
forge 'http://localhost'

mod 'puppetlabs/apache'
EOF

# Run librarian
librarian-puppet install --no-use-v1-api

Please post the errors or traces you get.

jskarpe commented 9 years ago

Librarian 0.1.2

root@ubuntu-1404:/tmp/my_module# librarian-puppet install --no-use-v1-api
ERROR: "librarian-puppet install" was called with arguments ["--no-use-v1-api"]
Usage: "librarian-puppet install"
root@ubuntu-1404:/tmp/my_module# librarian-puppet install --verbose
[Librarian] Ruby Version: 1.9.3
[Librarian] Ruby Platform: x86_64-linux
[Librarian] Rubygems Version: 1.8.23
[Librarian] Librarian Version: 0.1.1
[Librarian] Librarian Adapter: puppet
[Librarian] Librarian Adapter Version: 0.9.10
[Librarian] Project: /tmp/my_module
[Librarian] Specfile: Puppetfile
[Librarian] Lockfile: Puppetfile.lock
[Librarian] Git: /usr/bin/git
[Librarian] Git Version: 1.9.1
[Librarian] Git Environment Variables:
[Librarian]   (empty)
[Librarian] Pre-Cached Sources:
[Librarian] Post-Cached Sources:
[Librarian]   [:forge, "http://puppetforge.acme.com", {}]
[Librarian] Scheduling puppetlabs/concat (>= 0) <http://puppetforge.acme.com>
[Librarian] Scheduling puppetlabs/stdlib (>= 0) <http://puppetforge.acme.com>
[Librarian] Scheduling acme/bind (>= 0) <http://puppetforge.acme.com>
[Librarian] Resolving puppetlabs/concat (>= 0) <http://puppetforge.acme.com>
[Librarian]   Checking manifests
Unable to find module 'puppetlabs/concat' on http://puppetforge.acme.com
/usr/lib/ruby/vendor_ruby/librarian/puppet/source/forge.rb:22:in `versions'
/usr/lib/ruby/vendor_ruby/librarian/puppet/source/forge.rb:34:in `manifests'
/usr/lib/ruby/vendor_ruby/librarian/puppet/source/forge.rb:274:in `manifests'
/usr/lib/ruby/vendor_ruby/librarian/dependency.rb:113:in `cache_manifests!'
/usr/lib/ruby/vendor_ruby/librarian/dependency.rb:109:in `manifests'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:141:in `block in resolving_dependency_map_find_manifests'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:154:in `block (2 levels) in scope_resolving_dependency'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:168:in `block in scope_checking_manifests'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:208:in `scope'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:167:in `scope_checking_manifests'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:153:in `block in scope_resolving_dependency'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:208:in `scope'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:152:in `scope_resolving_dependency'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:140:in `resolving_dependency_map_find_manifests'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:56:in `recursive_resolve'
/usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:32:in `resolve'
/usr/lib/ruby/vendor_ruby/librarian/resolver.rb:16:in `resolve'
/usr/lib/ruby/vendor_ruby/librarian/action/resolve.rb:26:in `run'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:169:in `resolve!'
/usr/lib/ruby/vendor_ruby/librarian/puppet/cli.rb:63:in `install'
/usr/lib/ruby/vendor_ruby/thor/command.rb:27:in `run'
/usr/lib/ruby/vendor_ruby/thor/invocation.rb:121:in `invoke_command'
/usr/lib/ruby/vendor_ruby/thor.rb:363:in `dispatch'
/usr/lib/ruby/vendor_ruby/thor/base.rb:440:in `start'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:in `block (2 levels) in bin!'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:31:in `returning_status'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:in `block in bin!'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:47:in `with_environment'
/usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:in `bin!'
/usr/bin/librarian-puppet:9:in `<main>'
i11 commented 9 years ago

It seems like you're using very old version of librarian-puppet. It might have some API call differences. I would highly recommend stepping up closer to the latest version.

At the same time I can try debbuging a bit more. Could you please post the content of /opt/forge/modules and apache access log as well.

jskarpe commented 9 years ago

Upgraded to 2.0.1 now. Not much difference though

    root@ubuntu-1404:/tmp/mymodule# librarian-puppet install --no-use-v1-api --verbose
    [Librarian] Ruby Version: 1.9.3
    [Librarian] Ruby Platform: x86_64-linux
    [Librarian] Rubygems Version: 1.8.23
    [Librarian] Librarian Version: 0.1.2
    [Librarian] Librarian Adapter: puppet
    [Librarian] Librarian Adapter Version: 2.0.1
    [Librarian] Project: /tmp/mymodule
    [Librarian] Specfile: Puppetfile
    [Librarian] Lockfile: Puppetfile.lock
    [Librarian] Git: /usr/bin/git
    [Librarian] Git Version: 1.9.1
    [Librarian] Git Environment Variables:
    [Librarian]   (empty)
    [Librarian] Pre-Cached Sources:
    [Librarian] Post-Cached Sources:
    [Librarian]   [:forge, "http://puppetforge.acme.com", {}]
    [Librarian] Resolving puppetlabs-concat (>= 0) <http://puppetforge.acme.com>
    [Librarian]   Checking manifests
    Unable to find module 'puppetlabs-concat' on http://puppetforge.acme.com
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/source/forge/repo_v3.rb:42:in `get_module'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/source/forge/repo_v3.rb:19:in `get_versions'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/source/forge/repo.rb:15:in `versions'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/source/forge/repo.rb:42:in `manifests'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/source/forge.rb:152:in `manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/dependency.rb:117:in `cache_manifests!'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/dependency.rb:113:in `manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:152:in `block in resolving_dependency_map_find_manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:165:in `block (2 levels) in scope_resolving_dependency'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:179:in `block in scope_checking_manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:178:in `scope_checking_manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:164:in `block in scope_resolving_dependency'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:223:in `scope'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:163:in `scope_resolving_dependency'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:151:in `resolving_dependency_map_find_manifests'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:62:in `recursive_resolve'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver/implementation.rb:50:in `resolve'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/resolver.rb:23:in `resolve'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/action/resolve.rb:26:in `run'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:169:in `resolve!'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/lib/librarian/puppet/cli.rb:67:in `install'
    /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block (2 levels) in bin!'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:31:in `returning_status'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `block in bin!'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:47:in `with_environment'
    /var/lib/gems/1.9.1/gems/librarian-0.1.2/lib/librarian/cli.rb:26:in `bin!'
    /var/lib/gems/1.9.1/gems/librarian-puppet-2.0.1/bin/librarian-puppet:7:in `<top (required)>'
    /usr/local/bin/librarian-puppet:23:in `load'
    /usr/local/bin/librarian-puppet:23:in `<main>'
i11 commented 9 years ago

Weird. I would need to see apache access log and the content of the /opt/forge/modules to comment more.

jskarpe commented 9 years ago

The newer librarian successfully resolves the v3 API for puppetforge stuff, but the /opt/forge/modules issue still eludes me.

> ls /opt/forge/modules
acme-bind
> ls /opt/forge/modules/acme-bind
Gemfile       Modulefile  Rakefile   files  manifests      spec       tests
Gemfile.lock  README.md   build.xml  lib    metadata.json  templates
> curl http://localhost/v3/modules/acme-bind
{"errors":["404 Not found"]}

Is there anything in the modulefile.json or similar that needs to be updated?

access.log

[08/Dec/2014 15:14:17] "GET /v3/modules/acme-bind " 404 28 0.6836
i11 commented 9 years ago

Right. Here is the reason: forge expects tar.gz files in the modules directory.

Solution:

# Build the module
puppet module build /opt/forge/modules/acme-bind
mv /opt/forge/modules/acme-bind/pkg/*.tar.gz /opt/forge/modules

# Create Puppetfile
cat > Puppetfile <<EOF
forge 'http://localhost'

mod 'acme-bind'
EOF

# Run librarian
librarian-puppet install --no-use-v1-api --verbose

I have support for serving modules from source on the TODO list as well :)

The reason for it not finding puppetlabs-concat module is probably misconfiguration of proxy towards official forge. Did you do any more changes in config.ru compared to the updated comment 2?

jskarpe commented 9 years ago

Great success!

  1. Packaging a tarball solves the local module
  2. The puppet forge module was a problem with too old librarian. Upgrading librarian solved this issue, as it's now using the v3 urls.

Thanks!

i11 commented 9 years ago

Perfect. I'll add the passenger example into the readme as well. Thanks for taking all the pain of being first.

craiggenner commented 9 years ago

This no longer works as build requires 2 arguments: def build(backends, webui_root)

Any ideas how to fix this as :- run PuppetForgeServer::Server.new.build(backends, webui_root) doesn't work

i11 commented 9 years ago

Ah... totally forgot about this use-case. Try doing

run PuppetForgeServer::Server.new.build(backends, PuppetForgeServer::Utils::OptionParser.class_eval('@@DEFAULT_WEBUI_ROOT'))
craiggenner commented 9 years ago

No luck - Internal Server Error

and:

App 19324 stderr: [ 2015-06-18 11:23:48.6417 19351/0x000000025f2650(Worker 1) utils.rb:68 ]: *** Exception NameError in Rack application object (uninitialized constant Rack::Request::QUERY_STRING) (process 19351, thread 0x000000025f2650(Worker 1)):
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/request.rb:24:in `query_string'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/request.rb:188:in `GET'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/request.rb:230:in `params'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:901:in `call!'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:894:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/commonlogger.rb:33:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:218:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/logger.rb:15:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/commonlogger.rb:33:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:218:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:211:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-1.6.2/lib/rack/head.rb:13:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:181:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/sinatra-1.4.6/lib/sinatra/base.rb:2021:in `call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:152:in `block in call'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:96:in `block in recognize'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:68:in `optimized_each'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:95:in `recognize'
App 19324 stderr:       from /var/lib/gems/1.9.1/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:141:in `call'
App 19324 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
App 19324 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:142:in `accept_and_process_next_request'
App 19324 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
App 19324 stderr:       from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:448:in `block (3 levels) in start_threads'
i11 commented 9 years ago

Hmm.. weird, it worked for me with passenger 4.0.53 and apache 2.4.6. What kind of setup do you have? I could try to reproduce it.

craiggenner commented 9 years ago

apache 2.4.7-1ubuntu4.4 and passenger 4.0.37-2

This is a Ubuntu 14.04.2 LTS

i11 commented 9 years ago

I was able to reproduce it. Apparently rack gem has to be downgraded. I'll pin it in the gemspec and release a bit later today.

i11 commented 9 years ago

Try something like 1.4.5 if you're anxious.

craiggenner commented 9 years ago

Thanks, I'll wait until it has been released since I just end up fighting gem versions as soon as I try and use 1.4.5.

i11 commented 9 years ago

1.5.3 released. Please give it a go and let me know the results.

craiggenner commented 9 years ago

Thanks, that works.

I'll do a merge request to update the readme.

Appreciate the quick response on this, it really helps.

i11 commented 9 years ago

Sure thing. Glad I could help.