necrolyte2
commented
7 years ago We are seeing this same issue. 2.1.1 honored default sources (defined in /root.gemrc and /opt/chef/embedded/etc/gemrc) where 3.0.0 does not.
Closed chilicheech closed 4 years ago
necrolyte2
commented
7 years ago We are seeing this same issue. 2.1.1 honored default sources (defined in /root.gemrc and /opt/chef/embedded/etc/gemrc) where 3.0.0 does not.
1I1III1liL1
commented
7 years ago Still seeing this in our environment. Any ideas on ways around this @tas50 ?
rsilveira
commented
6 years ago up
thommay
commented
6 years ago This is working as intended.
Chef provides a configuration option, rubygems_url, that allows the admin to configure rubygems sources for the entire chef run in a consistent way, rather than having to use individual attributes on a per cookbook basis. See https://docs.chef.io/config_rb_client.html for more information.
cattywampus
commented
6 years ago @thommay It's my understanding that the rubygems_url feature was improved to support air gap issues starting with Chef Client 13, but this issue cites using Chef Client 12.15.19. I see the same issue using v12.19.36 and I even tried it with having the rubygems_url property set in the /etc/chef/client.rb configuration file.
I think the problem isn't necessarily with the chef-vault cookbook but rather how the chef client run behaves during the "Installing Cookbook Gems" step which occurs right after the cookbooks get synchronized to the server and before the compilation phase. This issue consistently reproduces itself when you specify a gem dependency in my cookbook's metadata.rb file, which the chef-vault cookbook does.
I would say this is more of an issue with Chef Client itself for v12 rather than this project. But if anyone has any other insights or opinions I would love to hear it. This is holding me up in a big way since other means of controlling the rubygems source list (i.e., root or system gemrc configuration) seems to have no impact on fixing this problem.
cattywampus
commented
6 years ago Update: @thommay was right about the rubygems_url. I had trouble using it at first because I'm producing this issue in Test Kitchen and I wasn't using the right process for setting the client settings for the chef_zero provisioner. To make this work, I added this to my .kitchen.yml provisioner:
provisioner:
name: chef_zero
client_rb:
rubygems_url: https://mygemserverIf you are using chef client, then the rubygems_url needs to go into your /etc/chef/client.rb configuration file on the node itself and should get worked into your bootstrap process.
haidangwa
commented
6 years ago So, is rubygems_url available in chef 12 and exactly which version? We're not ready to move to chef 13 yet, and our run lists consist of a mechanism to set the gem sources used by the chef-client. However, being that the gem dependency is in metadata.rb, our current mechanism doesn't quite work. Reverting to v2.1.1 is an option for us, which doesn't have this problem, btw.
cattywampus
commented
6 years ago It looks like this setting has existed since v12.8.1 but you might want to use at a minimum v12.10.48 as that appears to be the latest release in v12 where this was changed. We also had a step in our cookbook run list to set our gem sources but that doesn't work in this case because the recipes aren't being converged before the gems are being installed.
This only impacts cookbook runlists that include metadata.rb files which use the gem declaration. Which is why rolling back to v2.1.1 works, because the v3.0.0 release updated it's metadata to declare gem 'chef-vault'. Even if you roll back to v2.1.1, you'll likely encounter this problem as you bring in other community cookbooks. We discovered the same problem when we tried to use the systemd cookbook too.
The best option imo for moving forward is to set the rubygems_url setting in your /etc/chef/client.rb config file on the node.
ghost
commented
6 years ago I have run into this same problem, and find the gem metadata.rb approach somewhat limiting due to its inflexibility. I understand this isn't an issue with chef-vault, just airing my complaints in the hope that someone has advice.
Based on test kitchen runs:
bby-bishopclark
commented
6 years ago Or, how do we disable this bundler run (air-gap, gems versioned from scm) entirely?
username-is-already-taken2
commented
6 years ago I too have now come to the conclusion that I'm going to have to roll back this cookbook as its just not usable for me anymore, please maintiners not all machines have access to rubygems.org
I have a node which has the 3.3.0 version of the gem already installed, it has the `
"chef-vault": {
"version": "3.3.0"
},attribute set yet my chef run now fails as it attempts fetch metadata data rubygems.org which times out and fails.
as far as I can tell the only attribute that you now seem to make use of is the default['chef-vault']['databag_fallback'] so you should probably delete those from the attributes file and readme
bby-bishopclark
commented
6 years ago I suspect the code in question causing grief is actually at https://github.com/chef/chef/blob/master/lib/chef/cookbook/gem_installer.rb#L49 or so.
It happily automatically opens outside connections to compare gem versions from rubygems.org, before compile, so the leading solution is to bobbitt that code bit, or to light up and redundantly define a local gem mirror stub to satisfy the check. I found bobbitting the code worked well to let it compile and converge like before.
Um, any patch to make that permanent, though, should be promoted lightly. We ran into some egos so strong, so early-on, that we bailed on enterprise support and almost ditched chef. So if you pitch a bobbitting switch, go slow.
thommay
commented
6 years ago The chef vault gem is included in the latest stable chef-client release.
If your machine does not have access to rubygems but you have a local mirror, please set the configuration option rubygems_url in your client.rb to the url of your local rubygems mirror.
If you do not have a local rubygems mirror, we would be open to a discussion (either at https://feedback.chef.io or a bug report on https://github.com/chef/chef/issues/new ) on what the best user experience for disabling, globally, the installation of gems might be.
I'm fine with removing the gem installation part entirely.
chilicheech
commented
6 years ago Regarding the code mentioned by @bby-bishopclark here https://github.com/chef/chef/blob/master/lib/chef/cookbook/gem_installer.rb#L49 .. it would be awesome if it didn't hard code https://www.rubygems.org and instead read the sources from the Gem env in case the node has a gemrc so that it honors it.
Changing the client.rb is tricky because we need to run a converge to change it but we can't converge because chef-client fails to install the gem because the node doesn't have access to the interwebs.
Another thing to note is that it's doing a bundle install which is not capable of doing conservative installs, so it will upgrade dependent gems that are bundled with chef-client, even when those gems already satisfy the dependency requirements. The better thing to do IMHO is a gem install --file --conservative. gem install does not require a source in the Gemfile and it reads the source from the Gem env, so that kills 2 birds with 1 stone.
bby-bishopclark
commented
6 years ago I like @chilicheech's idea -- if indeed it will let the compilation continue, while still offering some downloads; I'd be happy if that entire bit of code were disabled and we had to chef_gem everything -- as long as we can manage that in cookbooks!
And "just go get everything you think you need from The World, whatever version you want" ruin consistency as well as security, which should more than rattle experienced security and build/promote types everywhere. But I was informed it's all On Me, so I'm not rejoining that discussion!
lamont-granquist
commented
6 years ago @chillicheech it would probably be fine to submit a PR to implement that patch. the use of where its easier to pre-configure a .gemrc on a box than it is to reconfigure config.rb is not anything that occurred to me when i wrote that. the goal of that code is clearly to allow airgapped environments to work correctly via config.rb, but if you can't modify config.rb when provisioning then you need to tell us what you can modify.
there's also the more global option of setting the default value of Chef::Config[:rubygems_url] based on the Gem.env as well, which seems like it would solve this problem and likely a lot more.
lamont-granquist
commented
4 years ago The changes to the gem metadata installer here got done. We're also going to be pulling chef-vault into core chef-client and won't be supporting the gem metadata installer any more. Also Chef 12 is EOL and no longer supported (as is Chef 13 and Chef 14 is getting its last update in a few days). There's no bug here around this cookbook in supported chef versions AFAIK.
Cookbook version
3.0.0
Chef-client version
12.15.19
Platform Details
RHEL 6
Scenario:
Run this cookbook in an air gapped environment with the
default['chef-vault']['gem_source']attribute set to an internal rubygems mirror.Steps to Reproduce:
Set the
default['chef-vault']['gem_source']attribute to an internal rubygems mirror. Run the cookbook in an environment that has no access torubygems.org.Expected Result:
The cookbook downloads and installs the gem from the internal rubygems mirror set by the
default['chef-vault']['gem_source']attribute.Actual Result:
The chef-client run fails because it can't talk to
rubygems.org.