Closed dfarrell07 closed 9 years ago
I'm still seeing this after a Vagrant update.
[~/puppet-opendaylight]$ vagrant --version
Vagrant 1.7.2
I'm still seeing F20 Beaker tests fail in the way described above:
Error: Could not enable opendaylight: Execution of '/sbin/chkconfig --add opendayl
ight' returned 1: error reading information on service opendaylight: No such file or direc
tory
Error: /Stage[main]/Opendaylight::Service/Service[opendaylight]/ensure: change fro
m stopped to running failed: Could not enable opendaylight: Execution of '/sbin/chkconfig
--add opendaylight' returned 1: error reading information on service opendaylight: No such
file or directory
After a BEAKER_destroy=no bundle exec rake fedora_20 &> fedora_20_1.log
, poking around the VM, it looks like ODL is installed and running.
[~/puppet-opendaylight/.vagrant/beaker_vagrant_files/fedora-20.yml]$ vagrant ssh
Last login: Mon Mar 2 02:03:51 2015 from 10.0.2.2
[vagrant@fedora-20 ~]$ ls /opt/
opendaylight-0.2.2 VBoxGuestAdditions-4.3.2
[vagrant@fedora-20 ~]$ ls /usr/lib/systemd/system | grep -i opend
opendaylight.service
[vagrant@fedora-20 ~]$ sudo systemctl status opendaylight
opendaylight.service - OpenDaylight SDN Controller
Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; disabled)
Active: active (running) since Mon 2015-03-02 02:08:54 UTC; 11min ago
Docs: https://wiki.opendaylight.org/view/Main_Page
http://www.opendaylight.org/
Process: 4122 ExecStart=/opt/opendaylight-0.2.2/bin/start (code=exited, status=0/SUCCESS)
Main PID: 4129 (java)
CGroup: /system.slice/opendaylight.service
└─4129 java -server -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+U...
Mar 02 02:08:54 fedora-20 systemd[1]: opendaylight.service: main process exited, cod...n/a
Mar 02 02:08:54 fedora-20 systemd[1]: Unit opendaylight.service entered failed state.
Mar 02 02:08:54 fedora-20 systemd[1]: Starting OpenDaylight SDN Controller...
Mar 02 02:08:54 fedora-20 systemd[1]: Started OpenDaylight SDN Controller.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@fedora-20 ~]$[~/puppet-opendaylight/.vagrant/beaker_vagrant_files/fedora-20.yml]$ vagrant ssh 21:10:51
Last login: Mon Mar 2 02:03:51 2015 from 10.0.2.2
[vagrant@fedora-20 ~]$ ls /opt/
opendaylight-0.2.2 VBoxGuestAdditions-4.3.2
[vagrant@fedora-20 ~]$ ls /usr/lib/systemd/system | grep -i opend
opendaylight.service
[vagrant@fedora-20 ~]$ sudo systemctl status opendaylight
opendaylight.service - OpenDaylight SDN Controller
Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; disabled)
Active: active (running) since Mon 2015-03-02 02:08:54 UTC; 11min ago
Docs: https://wiki.opendaylight.org/view/Main_Page
http://www.opendaylight.org/
Process: 4122 ExecStart=/opt/opendaylight-0.2.2/bin/start (code=exited, status=0/SUCCESS)
Main PID: 4129 (java)
CGroup: /system.slice/opendaylight.service
└─4129 java -server -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+U...
Mar 02 02:08:54 fedora-20 systemd[1]: opendaylight.service: main process exited, cod...n/a
Mar 02 02:08:54 fedora-20 systemd[1]: Unit opendaylight.service entered failed state.
Mar 02 02:08:54 fedora-20 systemd[1]: Starting OpenDaylight SDN Controller...
Mar 02 02:08:54 fedora-20 systemd[1]: Started OpenDaylight SDN Controller.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@fedora-20 ~]$
I am able to manually recreate the exit 1
described in the Beaker test stderr trace.
[vagrant@fedora-20 ~]$ /sbin/chkconfig --add opendaylight
error reading information on service opendaylight: No such file or directory
[vagrant@fedora-20 ~]$ echo $?
1
Making this Issue specific to F20, will handle F21 Beaker test failures in #63 (they are different failures).
Update: After fixing that^^ F21 fail, F21 started hitting this failure
Note that the Fedora 20 Puppet Vagrant VM provided by vagrant-opendaylight seems to work correctly. This issue is likely Beaker-specific, not a problem with the Puppet mod itself.
[~/vagrant-opendaylight]$ vagrant ssh fed20_puppet
Last login: Fri Dec 20 18:02:34 2013 from 10.0.2.2
[vagrant@localhost ~]$ ls
[vagrant@localhost ~]$ sudo systemctl status opendaylight
opendaylight.service - OpenDaylight SDN Controller
Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; enabled)
Active: active (running) since Mon 2015-03-02 21:15:16 UTC; 23s ago
Docs: https://wiki.opendaylight.org/view/Main_Page
http://www.opendaylight.org/
Main PID: 2152 (java)
CGroup: /system.slice/opendaylight.service
└─2152 java -server -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+U...
Mar 02 21:15:16 localhost.localdomain systemd[1]: Started OpenDaylight SDN Controller.
[vagrant@localhost ~]$ pgrep java
2152
[vagrant@localhost ~]$ cd /vagrant/
[vagrant@localhost vagrant]$ ./scripts/connect.sh
Will repeatedly attempt connecting to Karaf shell until it's ready
Warning: Permanently added '[localhost]:8101' (DSA) to the list of known hosts.
Authenticated with partial success.
________ ________ .__ .__ .__ __
\_____ \ ______ ____ ____ \______ \ _____ ___.__.| | |__| ____ | |___/ |_
/ | \\____ \_/ __ \ / \ | | \\__ \< | || | | |/ ___\| | \ __\
/ | \ |_> > ___/| | \| ` \/ __ \\___ || |_| / /_/ > Y \ |
\_______ / __/ \___ >___| /_______ (____ / ____||____/__\___ /|___| /__|
\/|__| \/ \/ \/ \/\/ /_____/ \/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight.
opendaylight-user@root>
I mentioned previously that I was able to replicate the exit 1
of chkconfig --add opendaylight
(seems to be the focus the error output) poking around a F20 VM provisioned by Beaker after a test.
I'm also able to replicate it on via vagrant-opendaylight on an F20 VM provisioned by this Puppet module's RPM install method.
[vagrant@localhost ~]$ chkconfig --add opendaylight
error reading information on service opendaylight: No such file or directory
[vagrant@localhost ~]$ echo $?
1
ODL does seem to be up and working.
[~/vagrant-opendaylight]$ vagrant ssh f20_pup_rpm
Last login: Fri Dec 20 18:02:34 2013 from 10.0.2.2
[vagrant@localhost ~]$ ls /opt/
opendaylight-0.2.2 VBoxGuestAdditions-4.3.2
[vagrant@localhost ~]$ sudo systemctl status opendaylight
opendaylight.service - OpenDaylight SDN Controller
Loaded: loaded (/usr/lib/systemd/system/opendaylight.service; enabled)
Active: active (running) since Tue 2015-03-03 21:57:41 UTC; 35s ago
Docs: https://wiki.opendaylight.org/view/Main_Page
http://www.opendaylight.org/
Main PID: 2175 (java)
CGroup: /system.slice/opendaylight.service
└─2175 java -server -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+U...
Mar 03 21:57:41 localhost.localdomain systemd[1]: Started OpenDaylight SDN Controller.
[vagrant@localhost ~]$ cd /vagrant/
[vagrant@localhost vagrant]$ ./scripts/connect.sh
# snip sshpass install
Will repeatedly attempt connecting to Karaf shell until it's ready
Warning: Permanently added '[localhost]:8101' (DSA) to the list of known hosts.
Authenticated with partial success.
________ ________ .__ .__ .__ __
\_____ \ ______ ____ ____ \______ \ _____ ___.__.| | |__| ____ | |___/ |_
/ | \\____ \_/ __ \ / \ | | \\__ \< | || | | |/ ___\| | \ __\
/ | \ |_> > ___/| | \| ` \/ __ \\___ || |_| / /_/ > Y \ |
\_______ / __/ \___ >___| /_______ (____ / ____||____/__\___ /|___| /__|
\/|__| \/ \/ \/ \/\/ /_____/ \/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight.
opendaylight-user@root>feature:list | grep openflowplugin-all
odl-openflowplugin-all | 0.0.5-Helium-SR2 | | openflowplugin-0.0.5-Helium-SR2 | OpenDaylight :: Openflow Plugin :: All
opendaylight-user@root>
Again, this this is the relevant part of the Beaker trace:
Error: Could not enable opendaylight: Execution of '/sbin/chkconfig --add opendayl
ight' returned 1: error reading information on service opendaylight: No such file or direc
tory
Error: /Stage[main]/Opendaylight::Service/Service[opendaylight]/ensure: change fro
m stopped to running failed: Could not enable opendaylight: Execution of '/sbin/chkconfig
--add opendaylight' returned 1: error reading information on service opendaylight: No such
file or directory
Digging through puppetlabs/puppet
's code, that comes from here:
def enable
chkconfig("--add", @resource[:name])
chkconfig(@resource[:name], :on)
rescue Puppet::ExecutionFailure => detail
raise Puppet::Error, "Could not enable #{self.name}: #{detail}", detail.backtrace
end
These docs from the top of that file are somewhat informative:
# Manage Red Hat services. Start/stop uses /sbin/service and enable/disable uses chkconfig
It seems that we either need to support that chkconfig
call or update Puppet's code to use systemd
.
I'm seeing this chkconfig
error in a CentOS 7 Puppet RPM [vagrant-opendaylight](# Manage Red Hat services. Start/stop uses /sbin/service and enable/disable uses chkconfig) VM as well:
[~/vagrant-opendaylight]$ vagrant ssh cent7_pup_rpm
Last login: Tue Jul 22 03:42:16 2014 from 10.0.2.2
[vagrant@localhost ~]$ chkconfig --add opendaylight
error reading information on service opendaylight: No such file or directory
Makes sense, as the enable
method mentioned above is on a type that is apparently used by default for all Red Hat family OSs:
defaultfor :osfamily => [:redhat, :suse]
I noticed that parallel to our lib/puppet/provider/service/redhat.rb
chkconfig-based enable
method, there's a lib/puppet/provider/service/systemd.rb
file with its own systemd-based enable
method. We should try to use this provider for Puppet's service
type, instead of the default redhat.rb
one.
def enable
output = systemctl("enable", @resource[:name])
rescue Puppet::ExecutionFailure
raise Puppet::Error, "Could not enable #{self.name}: #{output}", $!.backtrace
end
Given these relevant defaults in the systemd.rb
service
type provider:
defaultfor :osfamily => :redhat, :operatingsystemmajrelease => "7"
defaultfor :osfamily => :redhat, :operatingsystem => :fedora, :operatingsystemmajrelease => ["17", "18", "19", "20", "21"]
I would expect our F20 system to end up using this option, as it's a more specific match than the redhat.rb
one. That's not what seems to be happening in our Beaker F20 box however, as we're getting Execution of '/sbin/chkconfig --add opendaylight' returned 1:
for our #{detail}
value:
raise Puppet::Error, "Could not enable #{self.name}: #{detail}", detail.backtrace
And it seems that the only way Puppet would execute chkcommand --add
is via redhat.rb
:
[~/puppet]$ grep -rniI "chkconfig.*add" *
ext/redhat/puppet.spec.erb:311:/sbin/chkconfig --add puppet || :
ext/redhat/puppet.spec.erb:347:/sbin/chkconfig --add puppetmaster || :
lib/puppet/provider/service/redhat.rb:44: chkconfig("--add", @resource[:name])
spec/unit/provider/service/redhat_spec.rb:59: provider_class.expects(:chkconfig).with("--add", @resource[:name])
[~/puppet]$
Maybe somehow the Beaker tests are not getting the correct Facter facts for the F20 VM?
Looking at a vagrant-opendaylight
Fedora 20 Puppet RPM VM, manually checking facter
, I see the expected relevant values:
[~/vagrant-opendaylight]$ vagrant ssh f20_pup_rpm
Last login: Tue Mar 3 21:58:09 2015 from 10.0.2.2
[vagrant@localhost ~]$ facter | grep -i operatings
operatingsystem => Fedora
operatingsystemmajrelease => 20
operatingsystemrelease => 20
Again, those could trip either the redhat.rb
or systemd.rb
provider for the Puppet service
type, but I would have expected the more specific systemd.rb
one to be tripped.
Looking at a vagrant-opendaylight Fedora 20 Puppet RPM VM, manually checking facter, I see the expected relevant values:
Seeing the same expected values on a Beaker F20 Puppet RPM VM:
[~/puppet-opendaylight/.vagrant/beaker_vagrant_files/fedora-20.yml]$ vagrant ssh
Last login: Wed Mar 4 00:58:54 2015 from 10.0.2.2
[vagrant@fedora-20 ~]$ facter | grep -i operatings
operatingsystem => Fedora
operatingsystemmajrelease => 20
operatingsystemrelease => 20
[vagrant@fedora-20 ~]$
That VM failed the Beaker tests that were run when it was created with the same chkconfig --add
error:
Notice: /Stage[main]/Opendaylight::Config/File[org.apache.karaf.features.cfg]/content: con
tent changed '{md5}ebe3dbe5595f28c3b32b35b59e1d36cc' to '{md5}bf207ae2ade316ddc100da27488f
a722'
Info: Class[Opendaylight::Config]: Scheduling refresh of Class[Opendaylight::Service]
Info: Class[Opendaylight::Service]: Scheduling refresh of Service[opendaylight]
Error: Could not enable opendaylight: Execution of '/sbin/chkconfig --add opendaylight' returned 1: error reading information on service opendaylight: No such file or directory
Error: /Stage[main]/Opendaylight::Service/Service[opendaylight]/ensure: change from stopped to running failed: Could not enable opendaylight: Execution of '/sbin/chkconfig --add opendaylight' returned 1: error reading information on service opendaylight: No such file or directory
This seems to imply that the more specific rules for choosing a default Puppet service
type provider aren't being used, and that the one defined in redhat.rb
is being selected. Need to find the Puppet logic that makes this selection.
Grepping for default Puppet service
types used by various OSs, it seems like our situation with overlapping possible matches doesn't occur in any other case (except for suse
, which falls into both redhat.rb
and systemd.rb
like redhat
).
[~/puppet]$ grep -rniI "defaultfor" *
#snip
lib/puppet/provider/service/debian.rb:19: defaultfor :operatingsystem => :debian
lib/puppet/provider/service/freebsd.rb:6: defaultfor :operatingsystem => [:freebsd, :dragonfly]
lib/puppet/provider/service/launchd.rb:47: defaultfor :operatingsystem => :darwin
lib/puppet/provider/service/openbsd.rb:8: defaultfor :operatingsystem => :openbsd
lib/puppet/provider/service/openrc.rb:10: defaultfor :operatingsystem => :gentoo
lib/puppet/provider/service/openrc.rb:11: defaultfor :operatingsystem => :funtoo
lib/puppet/provider/service/openwrt.rb:9: defaultfor :operatingsystem => :openwrt
lib/puppet/provider/service/redhat.rb:11: defaultfor :osfamily => [:redhat, :suse]
lib/puppet/provider/service/smf.rb:15: defaultfor :osfamily => :solaris
lib/puppet/provider/service/src.rb:14: defaultfor :operatingsystem => :aix
lib/puppet/provider/service/systemd.rb:8: defaultfor :osfamily => [:archlinux]
lib/puppet/provider/service/systemd.rb:9: defaultfor :osfamily => :redhat, :operatingsystemmajrelease => "7"
lib/puppet/provider/service/systemd.rb:10: defaultfor :osfamily => :redhat, :operatingsystem => :fedora, :operatingsystemmajrelease => ["17", "18", "19", "20", "21"]
lib/puppet/provider/service/systemd.rb:11: defaultfor :osfamily => :suse, :operatingsystemmajrelease => ["12", "13"]
lib/puppet/provider/service/upstart.rb:21: defaultfor :operatingsystem => :ubuntu
lib/puppet/provider/service/windows.rb:14: defaultfor :operatingsystem => :windows
These unit tests should cover the Puppet behavior (choosing more specific default OSs matches) described above.
describe "when there are multiple defaultfor's with different specificity" do
before :each do
subject.defaultfor :operatingsystem => :os1
subject.defaultfor :operatingsystem => :os2, :operatingsystemmajrelease => "42"
end
let(:alternate) { type.provide(:alternate) {} }
it "should be default for a more specific, but matching, defaultfor" do
Facter.expects(:value).with(:operatingsystem).at_least_once.returns :os2
Facter.expects(:value).with(:operatingsystemmajrelease).at_least_once.returns "42"
expect(provider).to be_default
expect(alternate).not_to be_default
end
it "should be default for a less specific, but matching, defaultfor" do
Facter.expects(:value).with(:operatingsystem).at_least_once.returns :os1
expect(provider).to be_default
expect(alternate).not_to be_default
end
end
Note that I found them in Puppet's master
branch, but should confirm that they exist and pass for the version of Puppet I'm using.
On the failed F20 Beaker RPM box described above:
[vagrant@fedora-20 ~]$ puppet --version
3.7.4
This commit fixed the issue I'm seeing. Note that it's in master, but not in the 3.7.4 Puppet release.
[~/puppet]$ git log 3.7.4 master --no-merges
# snip
commit 1b8737d5a4f6f79ad28d38b65245b825e94a7006
Author: Scott Garman <scott.garman@puppetlabs.com>
Date: Fri Jan 30 14:17:27 2015 -0800
(PUP-3927) Ensure Fedora uses the systemd service provider
Recent changes to the redhat (chkconfig) service provider requires
features that the chkconfig shim doesn't support (specifically, --add).
Besides, the systemd provider should be the default for Fedora anyway.
Unfortunately, 3.7.4 is the latest Puppet release.
More git-fu to show that the relevant commit is in master but not 3.7.4:
[~/puppet]$ git show-branch --sha1-name master 3.7.4 | grep 1b8737d
+ [1b8737d] (PUP-3927) Ensure Fedora uses the systemd service provider
The +
in this context implies that in the diff of the refs, 1b8737d is in master
and not in 3.7.4
.
Here, I check out the 3.7.4 release tag and show that it was cut on Jan 26th 2015 while the commit we need was merged on January 30th 2015.
[~/puppet]$ git checkout tags/3.7.4
[~/puppet]$ git branch
* (detached from 3.7.4)
master
[~/puppet]$ git log --name-status HEAD~..HEAD | cat
commit 40e76e807f314e9435546f57013d66dd67e0d786
Author: Hailee Kenney <hailee@puppetlabs.com>
Date: Mon Jan 26 15:20:02 2015 -0800
(packaging) Update PUPPETVERSION to 3.7.4
M lib/puppet/version.rb
[~/puppet]$ git checkout master
[~/puppet]$ git show -s --format=medium 1b8737d5a4f6f79ad28d38b65245b825e94a7006 | cat
commit 1b8737d5a4f6f79ad28d38b65245b825e94a7006
Author: Scott Garman <scott.garman@puppetlabs.com>
Date: Fri Jan 30 14:17:27 2015 -0800
(PUP-3927) Ensure Fedora uses the systemd service provider
Recent changes to the redhat (chkconfig) service provider requires
features that the chkconfig shim doesn't support (specifically, --add).
Besides, the systemd provider should be the default for Fedora anyway.
[~/puppet]$
I labeled this as blocked
, as logged above. To be clear, it's blocked on Puppet cutting a new release that includes this fix. We're using Puppet 3.7.4 already, which is the latest version.
The fix we need was included when the Puppet 3.7.5 release was cut.
Hasn't been packaged for Fedora yet:
[~/puppet-opendaylight]$ puppet --version
3.7.4
[~/puppet-opendaylight]$ sudo yum update -y
[sudo] password for daniel:
Loaded plugins: langpacks, refresh-packagekit
bluejeans | 2.9 kB 00:00:00
google-chrome | 951 B 00:00:00
rpmfusion-free-updates | 3.3 kB 00:00:00
rpmfusion-nonfree-updates | 3.3 kB 00:00:00
updates/20/x86_64/metalink | 14 kB 00:00:00
No packages marked for update
Hasn't been packaged for Fedora yet:
Tuns out they use a different Yum repo. Once I pointed Yum at yum.puppetlabs.com I got the 3.7.5 update. Direct link to RPM.
[~]$ puppet --version 10:35:12
3.7.5
[~]$ sudo yum info puppet 10:35:14
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
Name : puppet
Arch : noarch
Version : 3.7.5
Release : 1.fc20
Size : 6.3 M
Repo : installed
From repo : puppetlabs-products
Summary : A network tool for managing many disparate systems
URL : http://puppetlabs.com
License : ASL 2.0
Description : Puppet lets you centrally manage every important aspect of your system using a
: cross-platform specification language that manages all the separate elements
: normally aggregated in different files, like users, cron jobs, and hosts,
: along with obviously discrete elements like packages, services, and files.
All tests passing after 1f7d340aaa74bd253994a1eb4456e39977401c3d.
Both the Fedora 20 and Fedora 21 Baker tests are failing with similar (I think the same) errors. CentOS 7 is working fine.