search

datagovuk / dgu-vagrant-puppet

[unmaintained] Container environment for booting fresh DGU virtual machines. Create VMs with Vagrant, configure them with Puppet.
37 stars 29 forks source link

open_uri.rb: redirection forbidden #109

Closed zvetaM closed 8 years ago

zvetaM commented 8 years ago

Hello.

When I reran setup few days ago, _install_puppetdependencies.sh started failing with the error


Querying Forge API for module puppetlabs/concat: http://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/concat
/usr/local/rvm/rubies/ruby-1.8.7-head/lib/ruby/1.8/open-uri.rb:174:in `open_loop': redirection forbidden: http://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/concat -> https://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/concat (RuntimeError)

I can reproduce the error with following commands

$ irb
1.8.7-p376 :002 > require 'open-uri'
 => true 
1.8.7-p376 :003 > open('http://github.com')
RuntimeError: redirection forbidden: http://github.com -> https://github.com/
            from /usr/local/rvm/rubies/ruby-1.8.7-head/lib/ruby/1.8/open-uri.rb:174:in `open_loop'
            from /usr/local/rvm/rubies/ruby-1.8.7-head/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
            from /usr/local/rvm/rubies/ruby-1.8.7-head/lib/ruby/1.8/open-uri.rb:518:in `open'
            from /usr/local/rvm/rubies/ruby-1.8.7-head/lib/ruby/1.8/open-uri.rb:30:in `open'
            from (irb):3

The gem _open_uriredirections solves the problem (see the proof bellow), but I do not know how to embed it into the setup...

$irb
1.8.7-p376 :001 > require 'open-uri'
         => true 
        1.8.7-p376 :002 > require 'rubygems'
         => true 
        1.8.7-p376 :003 > require 'open_uri_redirections'
         => true 
        1.8.7-p376 :004 > open('http://github.com', :allow_redirections => :safe)
 => #<File:/tmp/open-uri20160622-31895-sgkkig-0>
davidread commented 8 years ago

I don't know about allowing redirections. But I think there is another way - to just point it at the new url to start with. See https://github.com/datagovuk/dgu-vagrant-puppet/commit/3f6efac99e5ff6a7ed5f5f360ed942f79c685792#diff-e884420d180303a9808b5675600efbd7L3

BTW I've been doing a number of updates this week you may be interested in.

zvetaM commented 8 years ago

Dear David,

I've integrated the commit you mentioned in your last reply, and all later ones, into our installation script. But now the Puppet apply step fails with following error messages;

(they are the same for all pip install commands in init.pp)

Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns: 2016-06-29 14:18:58,322 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/bin/paster", line 9, in <module>
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     invoke(command, command_name, options, args[1:])
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     exit_code = runner.run(args)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     result = self.command()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckanext-archiver/ckanext/archiver/commands.py", line 110, in command
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     self._load_config()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/lib/cli.py", line 138, in _load_config
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     load_environment(conf.global_conf, conf.local_conf)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/config/environment.py", line 232, in load_environment
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     p.load_all(config)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/plugins/core.py", line 134, in load_all
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     load(*plugins)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/plugins/core.py", line 167, in load
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     plugins_update()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     environment.update_config()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckan/ckan/config/environment.py", line 357, in update_config
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     plugin.configure(config)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckanext-spatial/ckanext/spatial/plugin.py", line 74, in configure
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     setup_model()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/vagrant/pops-vagrant-puppet/src/ckanext-spatial/ckanext/spatial/model/package_extent.py", line 35, in setup
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     package_extent_table.create()
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 648, in create
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     checkfirst=checkfirst)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1555, in _run_visitor
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     conn._run_visitor(visitorcallable, element, **kwargs)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1202, in _run_visitor
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     **kwargs).traverse_single(element)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 119, in traverse_single
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     return meth(obj, **kw)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 722, in visit_table
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     _ddl_runner=self)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/sqlalchemy/event/attr.py", line 257, in __call__
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     fn(*args, **kw)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/geoalchemy/geometry.py", line 125, in before_create
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     self('before-create', target, connection)
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:   File "/home/co/ckan/lib/python2.7/site-packages/geoalchemy/geometry.py", line 108, in __call__
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns:     expression.ColumnCollection(*regular_cols))
Notice: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns: TypeError: __init__() takes exactly 1 argument (2 given)
Error: /home/co/ckan/bin/paster --plugin=ckanext-archiver archiver init --config=/var/ckan/ckan.ini returned 1 instead of one of [0]
Error: /Stage[main]/Dgu_ckan/Exec[paster archiver init]/returns: change from notrun to 0 failed: /home/co/ckan/bin/paster --plugin=ckanext-archiver archiver init --config=/var/ckan/ckan.ini returned 1 instead of one of [0]

A separate issue:

now that we have upgraded the version numbers of Ruby, Puppet and Librarian-Puppet, should we also upgrade the version number of highline gem? Currently we are still installing version 1.6.1 of Highline.

Thank you very much.

davidread commented 8 years ago

This error with ColumnCollection should be fixed with the newer version of SQLAlchemy - ensure you have 0.9.8, as per the init.pp, which I've just cherry-picked onto master.

Note I'll be merging the 14.04 branch soon. If you remain on 12.04 and there are any incompatibilities then you'll have to do that on your own branch I think.

I've no idea about Highline or what uses it - your advice on that is appreciated.