search

kindredgroup / puppet-forge-server

Private Puppet forge server supports local files and both v1 and v3 API proxies
69 stars 44 forks source link

Incorrect package obtained when using APIv3 #27

Closed bvis closed 9 years ago

bvis commented 9 years ago

Hi,

I have configured Puppet Forge 1.7.3 with local modules directory and using it as Puppetforge proxy as well. As far as I see this has not been corrected in the last version.

I've seen that when I execute the command to install this module "tPl0ch-composer" I obtain "puppetlabs-stdlib" instead. If I use the API v1 syntax "tPl0ch/composer" I get the right package

It can be seen in the home page when you search for this package, just try this URL in your Puppet-Forge-Server installation:

http://puppet-forge-server.localdomain/modules?query=tPl0ch-composer

And compare the answer with:

http://puppet-forge-server.localdomain/modules?query=tPl0ch/composer

You'll see that the first search returns you 2 results, the first in the list is incorrect, it's the "puppetlabs-stdlib" package.

i11 commented 9 years ago

Hi,

Thanks for reporting this! Unfortunately I was unable to reproduce this. Could you please provide a bit more details about the setup you have (OS, ruby version, puppet version, arguments used to start the forge server)? Also it would be nice if you could post exact commands you use.

Here is how I tried to run it:

bundle exec bin/puppet-forge-server -x https://forgeapi.puppetlabs.com
# First run
puppet module install tPl0ch-composer --module_repository http://localhost:8080 --debug

Notice: Preparing to install into /my/home/dir/.puppet/modules ...
Notice: Created target directory /my/home/dir/.puppet/modules
Notice: Downloading from http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/releases?module=tPl0ch-composer
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-git
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-vcsrepo
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-stdlib
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://localhost:8080/v3/files/v3/files/tPl0ch-composer-1.3.7.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/tPl0ch-composer20150828-28290-1nf9sgf | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Debug: HTTP GET http://localhost:8080/v3/files/v3/files/puppetlabs-git-0.4.0.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/puppetlabs-git20150828-28290-1p7c4mg | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Debug: HTTP GET http://localhost:8080/v3/files/v3/files/puppetlabs-stdlib-4.8.0.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/puppetlabs-stdlib20150828-28290-1lbhcjs | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Notice: Installing -- do not interrupt ...
/my/home/dir/.puppet/modules
└─┬ tPl0ch-composer (v1.3.7)
  └─┬ puppetlabs-git (v0.4.0)
    └── puppetlabs-stdlib (v4.8.0)

rm -rf ~/.puppet

# Consecutive run
puppet module install tPl0ch-composer --module_repository http://localhost:8080 --debug

Notice: Preparing to install into /my/home/dir/.puppet/modules ...
Notice: Created target directory /my/home/dir/.puppet/modules
Notice: Downloading from http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/releases?module=tPl0ch-composer
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-git
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-vcsrepo
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-stdlib
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://localhost:8080/v3/files/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c/t/tPl0ch-composer-1.3.7.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/tPl0ch-composer20150828-28070-x3btb | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Debug: HTTP GET http://localhost:8080/v3/files/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c/p/puppetlabs-git-0.4.0.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/puppetlabs-git20150828-28070-1pj3ohs | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Debug: HTTP GET http://localhost:8080/v3/files/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c/p/puppetlabs-stdlib-4.8.0.tar.gz
Debug: Executing 'gzip -dc /my/home/dir/.puppet/var/puppet-module/cache/puppetlabs-stdlib20150828-28070-1vwy5w | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Notice: Installing -- do not interrupt ...
/my/home/dir/.puppet/modules
└─┬ tPl0ch-composer (v1.3.7)
  └─┬ puppetlabs-git (v0.4.0)
    └── puppetlabs-stdlib (v4.8.0)

# Search with dash
puppet module search tPl0ch-composer --module_repository http://localhost:8080 --debug

Notice: Searching http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/modules?query=tPl0ch-composer
NAME                                                   DESCRIPTION                                                 AUTHOR                                         KEYWORDS                    
tPl0ch - Thomas Ploch <profiploch@gmail.com>-composer  This module provides the 'Composer' PHP dependency mana...  @tPl0ch - Thomas Ploch <profiploch@gmail.com>  php composer                

# Search with slash
puppet module search tPl0ch/composer --module_repository http://localhost:8080 --debug

Notice: Searching http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/modules?query=tPl0ch/composer
NAME                                                   DESCRIPTION                                                 AUTHOR                                         KEYWORDS                    
tPl0ch - Thomas Ploch <profiploch@gmail.com>-composer  This module provides the 'Composer' PHP dependency mana...  @tPl0ch - Thomas Ploch <profiploch@gmail.com>  php composer
bvis commented 9 years ago

It happens to me with 2 different puppet versions:

# puppet module search tPl0ch-composer --module_repository http://puppet-forge-server.localdomain --debug
Debug: Runtime environment: puppet_version=3.7.4, ruby_version=1.9.3, run_mode=user, default_encoding=UTF-8
Notice: Searching http://puppet-forge-server.localdomain ...
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/modules?query=tPl0ch-composer
NAME                                                   DESCRIPTION                                 AUTHOR                                         KEYWORDS

puppetlabs-stdlib                                      Standard library of resources for Puppe...  @puppetlabs                                    puppetlabs stdlib

tPl0ch - Thomas Ploch <profiploch@gmail.com>-composer  This module provides the 'Composer' PHP...  @tPl0ch - Thomas Ploch <profiploch@gmail.com>  php composer
# puppet module search tPl0ch-composer --module_repository http://puppet-forge-server.localdomain --debug
Debug: Runtime environment: puppet_version=4.2.1, ruby_version=1.9.3, run_mode=user, default_encoding=UTF-8
Notice: Searching http://puppet-forge-server.localdomain ...
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/modules?query=tPl0ch-composer
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
NAME                                                   DESCRIPTION                                 AUTHOR                                         KEYWORDS

puppetlabs-stdlib                                      Standard library of resources for Puppe...  @puppetlabs                                    puppetlabs stdlib

tPl0ch - Thomas Ploch <profiploch@gmail.com>-composer  This module provides the 'Composer' PHP...  @tPl0ch - Thomas Ploch <profiploch@gmail.com>  php composer

As you may see this is not happening with the slash instead:

# puppet module search tPl0ch/composer --module_repository http://puppet-forge-server.localdomain --debug
Debug: Runtime environment: puppet_version=4.2.1, ruby_version=1.9.3, run_mode=user, default_encoding=UTF-8
Notice: Searching http://puppet-forge-server.localdomain ...
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/modules?query=tPl0ch/composer
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
NAME                                                   DESCRIPTION                                 AUTHOR                                         KEYWORDS

tPl0ch - Thomas Ploch <profiploch@gmail.com>-composer  This module provides the 'Composer' PHP...  @tPl0ch - Thomas Ploch <profiploch@gmail.com>  php composer

I can see the same effect when doing a curl call to the GUI:

# curl http://puppet-forge-server.localdomain/modules?query=tPl0ch-composer | grep "<h3>"
                <h3>puppetlabs/stdlib</h3>
                <h3>tPl0ch - Thomas Ploch <profiploch@gmail.com>/composer</h3>
# curl http://puppet-forge-server.localdomain/modules?query=tPl0ch/composer | grep "<h3>"
                <h3>tPl0ch - Thomas Ploch <profiploch@gmail.com>/composer</h3>

The version installed of the server is: 1.7.3. Here you can see all the rubygems we have:

total 2.6M
drwxr-xr-x  2 root root  4.0K Aug  6 14:49 ./
drwx------ 27 marc users 4.0K Aug  6 14:46 ../
-rw-r--r--  1 root root  214K Aug  6 14:49 rubygem-backports-3.6.6-1.noarch.rpm
-rw-r--r--  1 root root   23K Aug  6 14:49 rubygem-deep_merge-1.0.1-1.noarch.rpm
-rw-r--r--  1 root root  230K Aug  6 14:49 rubygem-haml-4.0.6-1.noarch.rpm
-rw-r--r--  1 root root  532K Aug  6 14:49 rubygem-json-1.8.3-1.x86_64.rpm
-rw-r--r--  1 root root   42K Aug  6 14:49 rubygem-multi_json-1.11.2-1.noarch.rpm
-rw-r--r--  1 root root   35K Aug  6 14:49 rubygem-open4-1.3.4-1.noarch.rpm
-rw-r--r--  1 root root   16K Aug  6 14:49 rubygem-open_uri_redirections-0.2.1-1.noarch.rpm
-rw-r--r--  1 root root   68K Aug  6 14:49 rubygem-puppet-forge-server-1.7.3-1.noarch.rpm
-rw-r--r--  1 root root  435K Aug  6 14:49 rubygem-rack-1.5.5-1.noarch.rpm
-rw-r--r--  1 root root   49K Aug  6 14:49 rubygem-rack-mount-0.8.3-1.noarch.rpm
-rw-r--r--  1 root root   35K Aug  6 14:49 rubygem-rack-protection-1.5.3-1.noarch.rpm
-rw-r--r--  1 root root   47K Aug  6 14:49 rubygem-rack-test-0.6.3-1.noarch.rpm
-rw-r--r--  1 root root  701K Aug  6 14:49 rubygem-sinatra-1.4.6-1.noarch.rpm
-rw-r--r--  1 root root  109K Aug  6 14:49 rubygem-sinatra-contrib-1.4.6-1.noarch.rpm
-rw-r--r--  1 root root   91K Aug  6 14:49 rubygem-tilt-2.0.1-1.noarch.rpm

These are the arguments used to start the server, it's installed on a CentOS 7.0:

puppet-forge-server \
 --module-dir /var/lib/puppet-forge/modules \
 --proxy https://forgeapi.puppetlabs.com \
 --port 80 \
 --bind 0.0.0.0 \
 --cache-basedir /var/lib/puppet-forge/cache \
 --log-dir /var/log/puppet-forge \
 --daemonize

And it's true that using directly "install" instead of search it seems to work correctly:

# puppet module install tPl0ch-composer --module_repository http://puppet-forge-server.localdomain --debug
Debug: Runtime environment: puppet_version=4.2.1, ruby_version=1.9.3, run_mode=user, default_encoding=UTF-8
Notice: Preparing to install into /etc/puppetlabs/code/modules ...
Notice: Created target directory /etc/puppetlabs/code/modules
Notice: Downloading from http://puppet-forge-server.localdomain ...
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/releases?module=tPl0ch-composer
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/releases?module=puppetlabs-git
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/releases?module=puppetlabs-vcsrepo
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/releases?module=puppetlabs-stdlib
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/files/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c/t/tPl0ch-composer-1.3.7.tar.gz
Debug: Executing: 'gzip -dc /opt/puppetlabs/puppet/cache/puppet-module/cache/tPl0ch-composer20150901-21846-1n2upi7 | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 0:0 .'
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/files/v3/files/puppetlabs-git-0.4.0.tar.gz
Debug: Executing: 'gzip -dc /opt/puppetlabs/puppet/cache/puppet-module/cache/puppetlabs-git20150901-21846-1epzjpg | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 0:0 .'
Debug: HTTP GET http://puppet-forge-server.localdomain/v3/files/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c/p/puppetlabs-stdlib-4.8.0.tar.gz
Debug: Executing: 'gzip -dc /opt/puppetlabs/puppet/cache/puppet-module/cache/puppetlabs-stdlib20150901-21846-1x2nbvw | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 0:0 .'
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/modules
+-- tPl0ch-composer (v1.3.7)
  +-- puppetlabs-git (v0.4.0)
    +-- puppetlabs-stdlib (v4.8.0)

But as I'm using librarian-puppet for resolve all the dependencies, and it uses search to obtain the right versions of each library this bug is affecting me.

i11 commented 9 years ago

I did a few tests with both centos 7 and 6 and still no luck. I wonder if it freaks out because of something in the cache or modules dir. Could you please drop --daemonize and --log-dir arguments and post the output you get while running the search command? Also if you could clean up the cache and retry.

It would be ideal if you could create a Vagrantfile or Dockerfile that would replicate the issue. Here are my Vagranfiles I used to test: centos 7:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = '2'

puppet_version=ENV['PUPPET_VERSION'] || '3.7.4'
puppet_forge_server_version=ENV['PUPPET_FORGE_SERVER_VERSION'] || '1.7.3'

$script = <<SCRIPT
set -x
yum install -y ruby ruby-devel rubgems rubygems-devel git
gem install bundle --no-rdoc --no-ri
git clone https://github.com/unibet/puppet-forge-server.git
cd puppet-forge-server
git checkout #{puppet_forge_server_version}
sed -i "s/\\(spec.add_development_dependency 'rspec-core', '~> 3.1'\\)/\\1\\n  spec.add_development_dependency 'puppet', '#{puppet_version}'/" puppet-forge-server.gemspec
bundle install --path .bundle
bundle exec bin/puppet-forge-server -x https://forgeapi.puppetlabs.com 2>server.log &
sleep 5
bundle exec puppet module search tPl0ch-composer --module_repository http://localhost:8080 --debug
cat server.log
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = 'puppetlabs/centos-7.0-64-puppet'

  config.vm.provider "virtualbox" do |vb|
    vb.customize ["modifyvm", :id, "--memory",              "1024"]
    vb.customize ["modifyvm", :id, "--vram",                "32"]
    vb.customize ["modifyvm", :id, "--natdnsproxy1",        "off"]
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "off"]
  end

  %w(forge).each do |node_name|
    config.vm.define node_name do |node_config|
      node_config.vm.hostname = "#{node_name}.vagrant"
      node_config.vm.provision "shell", inline: $script
    end
  end
end

centos 6:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = '2'

puppet_version=ENV['PUPPET_VERSION'] || '3.7.4'
puppet_forge_server_version=ENV['PUPPET_FORGE_SERVER_VERSION'] || '1.7.3'

$script = <<SCRIPT
set -x
set -e
yum -y install ruby ruby-devel rubgems rubygems-devel git centos-release-SCL
yum -y install ruby193-ruby ruby193-ruby-devel ruby193-rubygems ruby193-rubygems-devel
scl enable ruby193 "gem install bundle --no-rdoc --no-ri"
git clone https://github.com/unibet/puppet-forge-server.git
cd puppet-forge-server
git checkout #{puppet_forge_server_version}
sed -i "s/\\(spec.add_development_dependency 'rspec-core', '~> 3.1'\\)/\\1\\n  spec.add_development_dependency 'puppet', '#{puppet_version}'/" puppet-forge-server.gemspec
scl enable ruby193 "/opt/rh/ruby193/root/usr/local/bin/bundle install --path .bundle"
scl enable ruby193 "/opt/rh/ruby193/root/usr/local/bin/bundle exec bin/puppet-forge-server -x https://forgeapi.puppetlabs.com 2>server.log &"
sleep 5
scl enable ruby193 "/opt/rh/ruby193/root/usr/local/bin/bundle exec puppet module search tPl0ch-composer --module_repository http://localhost:8080 --debug"
cat server.log
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = 'puppetlabs/centos-6.6-64-puppet'

  config.vm.provider "virtualbox" do |vb|
    vb.customize ["modifyvm", :id, "--memory",              "1024"]
    vb.customize ["modifyvm", :id, "--vram",                "32"]
    vb.customize ["modifyvm", :id, "--natdnsproxy1",        "off"]
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "off"]
  end

  %w(forge).each do |node_name|
    config.vm.define node_name do |node_config|
      node_config.vm.hostname = "#{node_name}.vagrant"
      node_config.vm.provision "shell", inline: $script
    end
  end
end

puppet librarian verbose output together with resulting server log would be interesting as well.

bvis commented 9 years ago

Oops, this error has disappeared after clean the cache!

This does not give me much confidence, because this error could occur again with no notice.

But as I'm not able to reproduce it by now I'll close this bug. If in the future I find it again I'll open another bug and I'll make a reference to this.

Thanks for your time.

i11 commented 9 years ago

I'll try to do some tests with cache as well a bit later. Hopefully I can find some clues what it could happen. I would also advise to update to 1.8.0 released yesterday. It contains a few fixes related to modules upload that could have mixed up directory backends.

Please don't hesitate to let me know if you find any problems. Thanks!

bvis commented 9 years ago

Thanks a lot, Ilja.