jfindlay
commented
9 years ago @dneades, thanks for the report.
Closed djneades closed 4 years ago
jfindlay
commented
9 years ago @dneades, thanks for the report.
cachedout
commented
9 years ago @dneades The code in question may be salt.grains.ssds._freebsd_ssds(). It's a fairly short function. Could you drop some debugging in there and see if you can figure out specifically what's slowing things down?
I am also going to cc: the original author of that code to see if he has any ideas. @kev009
djneades
commented
9 years ago @cachedout Thank you for the pointer. I’m looking at the code now and I can immediately the reason why kern.disks is being queried as an SSD. The output of sysctl kern.disks is kern.disks: da4 da3 da2 da1 da0, and every item in that string is tested to see whether it is an SSD. This problem is trivial to solve by changing the _freebsd_ssds() line:
for device in devices.split(' '):to
for device in devices.split(' ')[1:]:That change alone shaves about half a second off the execution time.
I’ll continue to have a look at this and see what else I can find.
djneades
commented
9 years ago @cachedout Some further information. Having fixed _freebsd_ssds() to avoid testing kern.disks:, execution time for the salt-call command is about 11.3 seconds. Changing _freebsd_ssds() to return an empty list immediately (i.e. so that it does no processing whatsoever) reduces total salt-call execution time to around 8.3 seconds.
So, _freebsd_ssds() is taking longer to execute than one might expect (see below), but apparently isn’t the main source of our excessive execution time. There is still a long pause after outputting the TRACE line telling me that arch cannot be found, and before the _freebsd_ssds() function is entered. The salt-call operation is therefore spending a long time before it even gets to the _freebsd_ssds() function.
It should be noted that, running directly from the command-line, sysctl kern.disks is very quick:
# time sysctl kern.disks
kern.disks: da4 da3 da2 da1 da0
0.00 real 0.00 user 0.00 sysAs are the camcontrol commands that the _freebsd_ssds() function is using:
# for disk in da4 da3 da2 da1 da0; do time camcontrol identify $disk; done |grep time
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed
0.02 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys
0.00 real 0.00 user 0.00 sys@cachedout Here’s some detailed logging. I’ve modified _freebsd_ssds() to log timings between most of its statements:
def _freebsd_ssds():
'''
Return a list of disk devices that ATA identify as SSDs (non-rotating)
'''
log.trace('{}: in _freebsd_ssds()'.format(datetime.utcnow()))
ssd_devices = []
SSD_TOKEN = 'non-rotating'
sysctl = salt.utils.which('sysctl')
camcontrol = salt.utils.which('camcontrol')
devices = __salt__['cmd.run']('{0} kern.disks'.format(sysctl))
for device in devices.split(' ')[1:]:
log.trace('{}: device = {}'.format(datetime.utcnow(), device))
identify = __salt__['cmd.run']('{0} identify {1}'.format(camcontrol,
device))
log.trace('{}: {} identify {} completed'.format(datetime.utcnow(), camcontrol, device))
if SSD_TOKEN in identify:
log.trace('{0}: Device {1} reports itself as an SSD'.format(datetime.utcnow(), device))
ssd_devices.append(device)
else:
log.trace('{0}: Device {1} does not report itself as an SSD'.format(datetime.utcnow(),
device))
log.trace('{}: leaving _freebsd_ssds()'.format(datetime.utcnow()))
return ssd_devicesHere is the output from salt-call:
# time salt-call -l all --local test.ping
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /usr/local/etc/salt/minion_id: <snip>
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /usr/local/etc/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': ['operator']
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'arch' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 2015-08-13 22:44:56.270446: in _freebsd_ssds()
[TRACE ] 2015-08-13 22:44:56.758706: device = da4
[TRACE ] 2015-08-13 22:44:57.269084: /sbin/camcontrol identify da4 completed
[TRACE ] 2015-08-13 22:44:57.269476: Device da4 does not report itself as an SSD
[TRACE ] 2015-08-13 22:44:57.269655: device = da3
[TRACE ] 2015-08-13 22:44:57.758654: /sbin/camcontrol identify da3 completed
[TRACE ] 2015-08-13 22:44:57.759039: Device da3 reports itself as an SSD
[TRACE ] 2015-08-13 22:44:57.759216: device = da2
[TRACE ] 2015-08-13 22:44:58.248208: /sbin/camcontrol identify da2 completed
[TRACE ] 2015-08-13 22:44:58.248591: Device da2 reports itself as an SSD
[TRACE ] 2015-08-13 22:44:58.248765: device = da1
[TRACE ] 2015-08-13 22:44:58.737951: /sbin/camcontrol identify da1 completed
[TRACE ] 2015-08-13 22:44:58.738335: Device da1 reports itself as an SSD
[TRACE ] 2015-08-13 22:44:58.738511: device = da0
[TRACE ] 2015-08-13 22:44:59.227717: /sbin/camcontrol identify da0 completed
[TRACE ] 2015-08-13 22:44:59.228105: Device da0 reports itself as an SSD
[TRACE ] 2015-08-13 22:44:59.228281: leaving _freebsd_ssds()
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/local/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x80ae79758> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'local': True}
local:
True
11.31 real 2.43 user 8.32 sysYou can see from this that the identify = __salt__['cmd.run']('{0} identify {1}'.format(camcontrol, device)) statement is taking about 0.5 s to execute each time it is called. That’s surprisingly slow given that the (apparently) same commands executed directly in a shell are virtually instantaneous. It seems likely that __salt__['cmd.run'] is adding substantial overhead in some way on this host.
I think that probably wraps-up all I can do on the _freebsd_ssds() function! Hope this is of some use.
djneades
commented
9 years ago (Heads-up: I have edited the previous three comments since I first made them.)
djneades
commented
9 years ago One further piece of information. I have a FreeBSD VM, in addition to the physical host to which all of the above information pertains. I am not seeing the same extreme slowness on that VM:
# uname -a
FreeBSD <snip> 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
# time salt-call -l all --local test.ping
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /usr/local/etc/salt/minion_id: <snip>
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /usr/local/etc/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': ['operator']
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'arch' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] Device kern.disks: does not report itself as an SSD
[TRACE ] Device da0 does not report itself as an SSD
[TRACE ] Device cd0 does not report itself as an SSD
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/local/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x80aea5758> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'local': True}
local:
True
2.19 real 0.63 user 0.91 sys
# salt --versions-report
Salt: 2015.5.3
Python: 2.7.10 (default, Aug 6 2015, 01:15:37)
Jinja2: 2.7.3
M2Crypto: 0.22
msgpack-python: 0.4.2
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.11
ioflo: Not Installed
PyZMQ: 14.6.0
RAET: Not Installed
ZMQ: 4.1.2
Mako: Not Installed
Tornado: Not InstalledNote however, that the test.ping still completes significantly faster if grains are skipped:
# time salt-call --local --skip-grains test.ping
local:
True
0.37 real 0.23 user 0.14 sysIt’s curious that the salt-call on the VM is so much faster than the physical host. Of course, the configurations are very different.
djneades
commented
9 years ago Some more digging has isolated the problem to a proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs) line in salt/modules/cmdmod.py.
I therefore added some print statements to the ctor of class TimedProc in salt/utils/timed_subprocess.py:
def __init__(self, args, **kwargs):
print('[PRINT ] {}: in TimedProc.__init__(self, args={}, kwargs={}'.format(datetime.utcnow(), args, kwargs))
self.command = args
self.stdin = kwargs.pop('stdin', None)
if self.stdin is not None:
# Translate a newline submitted as '\n' on the CLI to an actual
# newline character.
self.stdin = self.stdin.replace('\\n', '\n')
kwargs['stdin'] = subprocess.PIPE
self.with_communicate = kwargs.pop('with_communicate', True)
print('[PRINT ] {}: about to Popen...'.format(datetime.utcnow()))
self.process = subprocess.Popen(args, **kwargs)
print('[PRINT ] {}: leaving TimedProc.__init__(...)'.format(datetime.utcnow()))Running the salt-call again:
# time salt-call -l all --local test.ping
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /usr/local/etc/salt/minion_id: bhs-0.araxis.net
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /usr/local/etc/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': ['operator']
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'arch' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[PRINT ] 2015-08-14 10:39:09.557828: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.machine'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:09.557894: about to Popen...
[PRINT ] 2015-08-14 10:39:10.122935: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:10.123649: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.ncpu'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:10.123702: about to Popen...
[PRINT ] 2015-08-14 10:39:10.688838: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:10.689536: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:10.689587: about to Popen...
[PRINT ] 2015-08-14 10:39:11.255981: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:11.256938: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.physmem'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:11.256987: about to Popen...
[PRINT ] 2015-08-14 10:39:11.821944: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:11.822807: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.version'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:11.822855: about to Popen...
[PRINT ] 2015-08-14 10:39:12.387832: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:12.388536: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:12.388585: about to Popen...
[PRINT ] 2015-08-14 10:39:12.953633: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:12.954332: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.serial'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:12.954381: about to Popen...
[PRINT ] 2015-08-14 10:39:13.519428: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:13.520134: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.reldate'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:13.520184: about to Popen...
[PRINT ] 2015-08-14 10:39:14.085146: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:14.085841: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:14.085890: about to Popen...
[PRINT ] 2015-08-14 10:39:14.650844: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:14.651752: in TimedProc.__init__(self, args=['/usr/local/sbin/dmidecode'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -1, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:14.651802: about to Popen...
[PRINT ] 2015-08-14 10:39:15.217100: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:15.218786: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:15.218838: about to Popen...
[PRINT ] 2015-08-14 10:39:15.784154: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:15.784820: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:15.784870: about to Popen...
[PRINT ] 2015-08-14 10:39:16.349852: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:16.350544: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.hv_vendor'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:16.350593: about to Popen...
[PRINT ] 2015-08-14 10:39:16.915583: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:16.916294: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:16.916343: about to Popen...
[PRINT ] 2015-08-14 10:39:17.481963: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 10:39:17.482709: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'security.jail.jailed'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:17.482761: about to Popen...
[PRINT ] 2015-08-14 10:39:18.047788: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:18.048629: in _freebsd_ssds()
[PRINT ] 2015-08-14 10:39:18.049426: in TimedProc.__init__(self, args=['/sbin/sysctl', 'kern.disks'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:18.049472: about to Popen...
[PRINT ] 2015-08-14 10:39:18.614493: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:18.614913: device = da4
[PRINT ] 2015-08-14 10:39:18.615607: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da4'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:18.615656: about to Popen...
[PRINT ] 2015-08-14 10:39:19.180822: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:19.202765: /sbin/camcontrol identify da4 completed
[TRACE ] 2015-08-14 10:39:19.203153: Device da4 does not report itself as an SSD
[TRACE ] 2015-08-14 10:39:19.203334: device = da3
[PRINT ] 2015-08-14 10:39:19.203822: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da3'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:19.203869: about to Popen...
[PRINT ] 2015-08-14 10:39:19.769248: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:19.770908: /sbin/camcontrol identify da3 completed
[TRACE ] 2015-08-14 10:39:19.771289: Device da3 reports itself as an SSD
[TRACE ] 2015-08-14 10:39:19.771468: device = da2
[PRINT ] 2015-08-14 10:39:19.771948: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da2'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:19.771994: about to Popen...
[PRINT ] 2015-08-14 10:39:20.337096: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:20.338734: /sbin/camcontrol identify da2 completed
[TRACE ] 2015-08-14 10:39:20.339115: Device da2 reports itself as an SSD
[TRACE ] 2015-08-14 10:39:20.339293: device = da1
[PRINT ] 2015-08-14 10:39:20.339774: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da1'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:20.339820: about to Popen...
[PRINT ] 2015-08-14 10:39:20.904872: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:20.906514: /sbin/camcontrol identify da1 completed
[TRACE ] 2015-08-14 10:39:20.906900: Device da1 reports itself as an SSD
[TRACE ] 2015-08-14 10:39:20.907077: device = da0
[PRINT ] 2015-08-14 10:39:20.907555: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da0'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 10:39:20.907602: about to Popen...
[PRINT ] 2015-08-14 10:39:21.473954: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 10:39:21.475599: /sbin/camcontrol identify da0 completed
[TRACE ] 2015-08-14 10:39:21.476000: Device da0 reports itself as an SSD
[TRACE ] 2015-08-14 10:39:21.476180: leaving _freebsd_ssds()
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/local/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x80ae6e7d0> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'local': True}
local:
True
12.97 real 2.65 user 9.84 sysSo, surprisingly, it’s actually the self.process = subprocess.Popen(args, **kwargs) line that’s taking all the time – about half a second per invocation. We also see from where the big delay prior to entering _freebsd_ssds() arises.
Finally, note that I have upgraded my machine to FreeBSD 10.2. The timings seem a little worse with this, but otherwise the problem is as previously characterized.
djneades
commented
9 years ago I wrote test1.py to isolate the problem in a standalone test case:
import subprocess
from datetime import datetime
args = ['/sbin/sysctl', '-n', 'hw.machine']
kwargs = kwargs={
'with_communicate': True,
'shell': False,
'stderr': -2,
'stdout': -1,
'close_fds': True,
'stdin': None,
'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/etc', 'SUDO_UID': '1\
001', 'TERM': 'xterm-256color', 'PWD': '/etc/pkg', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/g\
ames:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'},
'cwd': '/root'}
stdin = kwargs.pop('stdin', None)
with_communicate = kwargs.pop('with_communicate', True)
print('[PRINT ] {}: about to Popen...'.format(datetime.utcnow()))
process = subprocess.Popen(args, **kwargs)
print('[PRINT ] {}: done'.format(datetime.utcnow()))This reproduces the problem:
# time python2.7 test1.py
[PRINT ] 2015-08-14 10:51:08.177299: about to Popen...
[PRINT ] 2015-08-14 10:51:08.741294: done
amd64
0.58 real 0.10 user 0.47 sysA little experimentation shows that the delay is entirely due to the presence of the 'close_fds': True kwarg. With that line commented out:
# time python2.7 test1.py
[PRINT ] 2015-08-14 10:55:11.154827: about to Popen...
[PRINT ] 2015-08-14 10:55:11.155834: done
0.01 real 0.00 user 0.00 sysI’m not sure why the file descriptors are being closed by the various calls to salt.utils.timed_subprocess.TimedProc (though I’m guessing this is related to security), nor why this is taking so long on my physical host.
Running the same test program on my FreeBSD 10.1 VM shows again that, although problem is not as severe, closing the file descriptors still takes a significant amount of time. First, here is a run on my VM with the 'close_fds': True kwarg present:
# time python2.7 test1.py
[PRINT ] 2015-08-14 10:59:36.485904: about to Popen...
[PRINT ] 2015-08-14 10:59:36.542353: done
0.07 real 0.01 user 0.05 sysAnd here is a run with the close_fds line commented out:
# time python2.7 test1.py
[PRINT ] 2015-08-14 11:00:28.631821: about to Popen...
[PRINT ] 2015-08-14 11:00:28.632835: done
0.01 real 0.00 user 0.00 sysBack on my physical host, I wished to double-check my findings. I therefore added kwargs.pop('close_fds') (without regard for any side effects!) after the self.stdin line in the TimedProc ctor. This did indeed solve the slow execution problem:
# time salt-call -l all --local test.ping
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /usr/local/etc/salt/minion_id: bhs-0.araxis.net
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /usr/local/etc/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': ['operator']
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'arch' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[PRINT ] 2015-08-14 11:07:29.960647: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.machine'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.960712: about to Popen...
[PRINT ] 2015-08-14 11:07:29.962764: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.963497: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.ncpu'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.963547: about to Popen...
[PRINT ] 2015-08-14 11:07:29.965546: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.966255: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.966307: about to Popen...
[PRINT ] 2015-08-14 11:07:29.968318: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.969282: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.physmem'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.969330: about to Popen...
[PRINT ] 2015-08-14 11:07:29.971336: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.972193: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.version'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.972242: about to Popen...
[PRINT ] 2015-08-14 11:07:29.974251: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.974967: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.975018: about to Popen...
[PRINT ] 2015-08-14 11:07:29.977038: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.977747: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.serial'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.977798: about to Popen...
[PRINT ] 2015-08-14 11:07:29.979829: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.980526: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.reldate'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.980581: about to Popen...
[PRINT ] 2015-08-14 11:07:29.982582: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.983278: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.983329: about to Popen...
[PRINT ] 2015-08-14 11:07:29.985349: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.986262: in TimedProc.__init__(self, args=['/usr/local/sbin/dmidecode'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -1, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.986313: about to Popen...
[PRINT ] 2015-08-14 11:07:29.988390: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.990096: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.990146: about to Popen...
[PRINT ] 2015-08-14 11:07:29.992225: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.992906: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.992958: about to Popen...
[PRINT ] 2015-08-14 11:07:29.994994: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.995691: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.hv_vendor'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.995742: about to Popen...
[PRINT ] 2015-08-14 11:07:29.997773: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:29.998493: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:29.998544: about to Popen...
[PRINT ] 2015-08-14 11:07:30.000550: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:07:30.001293: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'security.jail.jailed'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.001346: about to Popen...
[PRINT ] 2015-08-14 11:07:30.003361: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.004193: in _freebsd_ssds()
[PRINT ] 2015-08-14 11:07:30.004941: in TimedProc.__init__(self, args=['/sbin/sysctl', 'kern.disks'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.004989: about to Popen...
[PRINT ] 2015-08-14 11:07:30.007059: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.007488: device = da4
[PRINT ] 2015-08-14 11:07:30.008190: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da4'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.008242: about to Popen...
[PRINT ] 2015-08-14 11:07:30.010343: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.032592: /sbin/camcontrol identify da4 completed
[TRACE ] 2015-08-14 11:07:30.032987: Device da4 does not report itself as an SSD
[TRACE ] 2015-08-14 11:07:30.033167: device = da3
[PRINT ] 2015-08-14 11:07:30.033657: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da3'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.033704: about to Popen...
[PRINT ] 2015-08-14 11:07:30.035771: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.037450: /sbin/camcontrol identify da3 completed
[TRACE ] 2015-08-14 11:07:30.037839: Device da3 reports itself as an SSD
[TRACE ] 2015-08-14 11:07:30.038017: device = da2
[PRINT ] 2015-08-14 11:07:30.038505: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da2'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.038550: about to Popen...
[PRINT ] 2015-08-14 11:07:30.040623: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.042285: /sbin/camcontrol identify da2 completed
[TRACE ] 2015-08-14 11:07:30.042672: Device da2 reports itself as an SSD
[TRACE ] 2015-08-14 11:07:30.042849: device = da1
[PRINT ] 2015-08-14 11:07:30.043329: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da1'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.043374: about to Popen...
[PRINT ] 2015-08-14 11:07:30.045457: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.047130: /sbin/camcontrol identify da1 completed
[TRACE ] 2015-08-14 11:07:30.047515: Device da1 reports itself as an SSD
[TRACE ] 2015-08-14 11:07:30.047693: device = da0
[PRINT ] 2015-08-14 11:07:30.048176: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da0'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:07:30.048221: about to Popen...
[PRINT ] 2015-08-14 11:07:30.050337: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:07:30.051993: /sbin/camcontrol identify da0 completed
[TRACE ] 2015-08-14 11:07:30.052377: Device da0 reports itself as an SSD
[TRACE ] 2015-08-14 11:07:30.052554: leaving _freebsd_ssds()
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/local/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x80ae79758> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'local': True}
local:
True
1.25 real 0.34 user 0.33 sysHere is normal execution without the kwargs.pop('close_fds') line, for comparison:
# time salt-call -l all --local test.ping
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /usr/local/etc/salt/minion_id: bhs-0.araxis.net
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /usr/local/etc/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': ['operator']
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion
[DEBUG ] Including configuration from '/usr/local/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /usr/local/etc/salt/minion.d/_schedule.conf
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'ip' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[TRACE ] 'arch' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/games', '/usr/local/sbin', '/usr/local/bin', '/home/djn/bin']
[PRINT ] 2015-08-14 11:14:16.959044: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.machine'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:16.959109: about to Popen...
[PRINT ] 2015-08-14 11:14:17.524523: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:17.525252: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.ncpu'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:17.525303: about to Popen...
[PRINT ] 2015-08-14 11:14:18.090558: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:18.091263: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:18.091312: about to Popen...
[PRINT ] 2015-08-14 11:14:18.656574: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:18.657541: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'hw.physmem'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:18.657590: about to Popen...
[PRINT ] 2015-08-14 11:14:19.223054: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:19.223916: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.version'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:19.223965: about to Popen...
[PRINT ] 2015-08-14 11:14:19.788989: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:19.789700: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:19.789752: about to Popen...
[PRINT ] 2015-08-14 11:14:20.355132: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:20.355848: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.serial'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:20.355899: about to Popen...
[PRINT ] 2015-08-14 11:14:20.922562: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:20.923272: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.bios.reldate'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:20.923325: about to Popen...
[PRINT ] 2015-08-14 11:14:21.488518: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:21.489224: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:21.489275: about to Popen...
[PRINT ] 2015-08-14 11:14:22.054275: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:22.055197: in TimedProc.__init__(self, args=['/usr/local/sbin/dmidecode'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -1, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:22.055248: about to Popen...
[PRINT ] 2015-08-14 11:14:22.620372: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:22.622080: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.product'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:22.622130: about to Popen...
[PRINT ] 2015-08-14 11:14:23.187247: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:23.187926: in TimedProc.__init__(self, args=['/bin/kenv', 'smbios.system.maker'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:23.187976: about to Popen...
[PRINT ] 2015-08-14 11:14:23.752948: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:23.753649: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.hv_vendor'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:23.753700: about to Popen...
[PRINT ] 2015-08-14 11:14:24.318749: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:24.319471: in TimedProc.__init__(self, args=['/sbin/sysctl', 'hw.model'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:24.319521: about to Popen...
[PRINT ] 2015-08-14 11:14:24.884502: leaving TimedProc.__init__(...)
[PRINT ] 2015-08-14 11:14:24.885253: in TimedProc.__init__(self, args=['/sbin/sysctl', '-n', 'security.jail.jailed'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:24.885305: about to Popen...
[PRINT ] 2015-08-14 11:14:25.450475: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:25.451314: in _freebsd_ssds()
[PRINT ] 2015-08-14 11:14:25.452103: in TimedProc.__init__(self, args=['/sbin/sysctl', 'kern.disks'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:25.452151: about to Popen...
[PRINT ] 2015-08-14 11:14:26.018681: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:26.019110: device = da4
[PRINT ] 2015-08-14 11:14:26.019853: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da4'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:26.019906: about to Popen...
[PRINT ] 2015-08-14 11:14:26.585323: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:26.607613: /sbin/camcontrol identify da4 completed
[TRACE ] 2015-08-14 11:14:26.608008: Device da4 does not report itself as an SSD
[TRACE ] 2015-08-14 11:14:26.608189: device = da3
[PRINT ] 2015-08-14 11:14:26.608684: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da3'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:26.608730: about to Popen...
[PRINT ] 2015-08-14 11:14:27.173738: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:27.175412: /sbin/camcontrol identify da3 completed
[TRACE ] 2015-08-14 11:14:27.175802: Device da3 reports itself as an SSD
[TRACE ] 2015-08-14 11:14:27.175980: device = da2
[PRINT ] 2015-08-14 11:14:27.176464: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da2'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:27.176509: about to Popen...
[PRINT ] 2015-08-14 11:14:27.741576: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:27.743228: /sbin/camcontrol identify da2 completed
[TRACE ] 2015-08-14 11:14:27.743616: Device da2 reports itself as an SSD
[TRACE ] 2015-08-14 11:14:27.743793: device = da1
[PRINT ] 2015-08-14 11:14:27.744279: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da1'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:27.744324: about to Popen...
[PRINT ] 2015-08-14 11:14:28.309500: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:28.311163: /sbin/camcontrol identify da1 completed
[TRACE ] 2015-08-14 11:14:28.311550: Device da1 reports itself as an SSD
[TRACE ] 2015-08-14 11:14:28.311729: device = da0
[PRINT ] 2015-08-14 11:14:28.312216: in TimedProc.__init__(self, args=['/sbin/camcontrol', 'identify', 'da0'], kwargs={'with_communicate': True, 'shell': False, 'stderr': -2, 'stdout': -1, 'close_fds': True, 'stdin': None, 'env': {'USERNAME': 'root', 'SUDO_COMMAND': '/bin/sh', 'SUDO_GID': '1001', 'SHELL': '/bin/sh', 'OLDPWD': '/root', 'SUDO_UID': '1001', 'TERM': 'xterm-256color', 'PWD': '/home/djn', 'LOGNAME': 'root', 'USER': 'root', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/djn/bin', 'MAIL': '/var/mail/root', 'SUDO_USER': 'djn', 'HOME': '/root', 'LC_ALL': 'C'}, 'cwd': '/root'}
[PRINT ] 2015-08-14 11:14:28.312261: about to Popen...
[PRINT ] 2015-08-14 11:14:28.877281: leaving TimedProc.__init__(...)
[TRACE ] 2015-08-14 11:14:28.878943: /sbin/camcontrol identify da0 completed
[TRACE ] 2015-08-14 11:14:28.879327: Device da0 reports itself as an SSD
[TRACE ] 2015-08-14 11:14:28.879505: leaving _freebsd_ssds()
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/local/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x80ae787d0> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'local': True}
local:
True
12.98 real 2.86 user 9.64 sysA Google search for ‘subprocess close_fds slow’ turns up a number of interesting results…
Based on some of the discussions listed in those search results about Python’s handling of close_fds, I wondered whether limiting the number of file descriptors that can be opened by a process would make a difference. Indeed it does:
# ulimit -n
3773106
# time salt-call --local test.ping
local:
True
12.96 real 2.77 user 9.71 sys
# ulimit -n 8192
# ulimit -n
8192
# time salt-call --local test.ping
local:
True
1.18 real 0.42 user 0.27 sysJust to make really sure that we are definitely controlling the problem, let’s set ulimit -n high again:
# ulimit -n 3000000
# time salt-call --local test.ping
local:
True
15.89 real 2.10 user 7.96 sysOuch!
This nicely explains why my VM was performing so much better than the physical host. Here are the relevant sysctl values on the VM:
# sysctl -a|grep maxfiles
kern.maxfiles: 523432
kern.maxfilesperproc: 471087And on my physical host:
# sysctl -a|grep maxfiles
kern.maxfiles: 4192348
kern.maxfilesperproc: 3773106The primary actual problem here is that, because of the 'close_fds': True kwarg, the various command invocations via TimedProcess can be exceedingly slow on machines where the default ulimit -n is high, as it is on my 128 GB RAM physical host. The impact of this on general salt performance is potentially wide-ranging and significant.
Since the slow Python close_fds behaviour does not seem to be OS-specific, this problem may also affect Linux hosts with high ulimit -n values, in addition to FreeBSD ones.
(Incidentally, the issue labels probably need tweaking, since this turns out not actually to be a grains-specific problem.)
Finally, there is also the more minor secondary issue of _freebsd_ssds() attempting to query a non-device kern.disks: to see whether it is an SSD, and that should definitely be fixed. Fortunately, the fix is trivial.
kev009
commented
9 years ago That's a nice analysis @dneades! Here's a PR to fix the kern.disks issue you pointed out: https://github.com/saltstack/salt/pull/26700
I don't have any insight into the general issue with close_fds. I haven't noticed yet because most our interactive salt use is in Vagrant VMs and the big boxes are schedule driven.
djneades
commented
9 years ago Thank you for the PR @kev009, much appreciated.
With respect to the close_fds issue, one solution would be to add a minion configuration parameter akin to the master’s max_open_files, with a sensible default (say, 2048). That would apply a ulimit -n (or equivalent) to any salt minion processes. Such a limit would obviously need to work with both the salt minion daemon and invocations via salt-ssh.
Karunamon
commented
9 years ago I've got a very similar-sounding issue on 2015.5.5 on a masterless setup on Fedora, where grain collection is taking ages, and sure enough, most of that time is spent checking if my drives are SSDs.
This is also a single state sls that's only mission in life is to install vim.
The difference is, I don't have a high ulimit - it's 1024, on pretty weak hardware.
Salt: 2015.5.5
Python: 2.7.10 (default, Sep 8 2015, 17:20:17)
Jinja2: 2.8
M2Crypto: 0.21.1
msgpack-python: 0.4.6
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.11
ioflo: Not Installed
PyZMQ: 14.7.0
RAET: Not Installed
ZMQ: 4.1.2
Mako: Not Installed
Tornado: Not Installed
timelib: Not Installed
dateutil: Not InstalledAnd the trace log:
sudo time salt-call --config-dir salt/ state.highstate test=true -l trace
[DEBUG ] Reading configuration from /home/tk/dotfiles/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: shiva.tkware.local
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found the in the configuration. Not loading the Logstash logging handlers module.
[DEBUG ] Configuration file path: /home/tk/dotfiles/salt/minion
[TRACE ] Trying pysss.getgrouplist for 'root'
[TRACE ] Trying generic group list for 'root'
[TRACE ] Group list for user 'root': []
[DEBUG ] Reading configuration from /home/tk/dotfiles/salt/minion
### Sits here for about 40 seconds ###
[TRACE ] Device sda reports itself as an SSD
[TRACE ] Device dm-0 reports itself as an SSD
[TRACE ] Device dm-1 reports itself as an SSD
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded state.highstate
[DEBUG ] LazyLoaded grains.get
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init
[DEBUG ] Updating roots fileserver cache
[TRACE ] Lockfile /var/cache/salt/minion/file_lists/roots/.base.w created
[TRACE ] Lockfile /var/cache/salt/minion/file_lists/roots/.base.w removed
[DEBUG ] Reading configuration from /home/tk/dotfiles/salt/minion
### Sits here for another 30 seconds ###
[TRACE ] Device sda reports itself as an SSD
[TRACE ] Device dm-0 reports itself as an SSD
[TRACE ] Device dm-1 reports itself as an SSD
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://top.sls'
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:
base:
'*':
- packages
[DEBUG ] LazyLoaded config.get
[DEBUG ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['packages'])]))])
[DEBUG ] LazyLoaded saltutil.sync_all
[DEBUG ] Syncing all
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[INFO ] Syncing beacons for environment 'base'
[INFO ] Loading cache from salt://_beacons, for base)
[DEBUG ] LazyLoaded cp.cache_dir
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] Could not LazyLoad roots.init
[DEBUG ] Updating roots fileserver cache
[INFO ] Caching directory '_beacons/' for environment 'base'
[TRACE ] Lockfile /var/cache/salt/minion/file_lists/roots/.base.w created
[TRACE ] Lockfile /var/cache/salt/minion/file_lists/roots/.base.w removed
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_beacons'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO ] Syncing modules for environment 'base'
[INFO ] Loading cache from salt://_modules, for base)
[INFO ] Caching directory '_modules/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_modules'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO ] Syncing states for environment 'base'
[INFO ] Loading cache from salt://_states, for base)
[INFO ] Caching directory '_states/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_states'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base)
[INFO ] Caching directory '_grains/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_grains'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO ] Syncing renderers for environment 'base'
[INFO ] Loading cache from salt://_renderers, for base)
[INFO ] Caching directory '_renderers/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_renderers'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO ] Syncing returners for environment 'base'
[INFO ] Loading cache from salt://_returners, for base)
[INFO ] Caching directory '_returners/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_returners'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/outputters'
[INFO ] Syncing outputters for environment 'base'
[INFO ] Loading cache from salt://_outputters, for base)
[INFO ] Caching directory '_outputters/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_outputters'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO ] Syncing utils for environment 'base'
[INFO ] Loading cache from salt://_utils, for base)
[INFO ] Caching directory '_utils/' for environment 'base'
[TRACE ] Returning file_lists cache data from /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_utils'
[DEBUG ] Refreshing modules...
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded saltutil.refresh_modules
[DEBUG ] LazyLoaded event.fire
[DEBUG ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_11309127e0_pub.ipc
[DEBUG ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_11309127e0_pull.ipc
[DEBUG ] Sending event - data = {'_stamp': '2015-11-20T00:39:17.374357'}
[DEBUG ] Sending event - data = {'_stamp': '2015-11-20T00:39:18.376258'}
[INFO ] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://packages.sls'
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/packages.sls:
vim:
pkg:
- name: 'vim-enhanced'
- installed
[DEBUG ] LazyLoaded config.get
[DEBUG ] Results of YAML rendering:
OrderedDict([('vim', OrderedDict([('pkg', [OrderedDict([('name', 'vim-enhanced')]), 'installed'])]))])
[TRACE ] Loaded yumpkg as virtual pkg
[DEBUG ] LazyLoaded pkg.install
[DEBUG ] LazyLoaded pkg.installed
[TRACE ] Loaded localemod as virtual locale
[TRACE ] Loaded groupadd as virtual group
[TRACE ] Loaded systemd as virtual service
[TRACE ] 'sysbench' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'psql' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded linux_sysctl as virtual sysctl
[TRACE ] 'ipset' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'initctl' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'cpan' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded augeas_cfg as virtual augeas
[TRACE ] 'rabbitmqctl' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'zpool' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'npm' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[DEBUG ] Error loading module.npm: npm execution module could not be loaded because the npm binary could not be located
[TRACE ] Loaded gnomedesktop as virtual gnome
[TRACE ] 'chef-client' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'gluster' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'svn' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'system_profiler' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'uwsgi' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'varnishd' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Depends decorator instantiated with dep list of ('non_existantmodulename',)
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'sensors' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'solr' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'apache-solr' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[DEBUG ] Error loading module.tls: ['PyOpenSSL version 0.10 or later must be installed before this module can be used.']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'guestmount' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'hponcfg' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'lxc-start' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded parted as virtual partition
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[DEBUG ] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'facter' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded sysmod as virtual sys
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded rpm as virtual lowpkg
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded zcbuildout as virtual buildout
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'nft' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded mod_random as virtual random
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded htpasswd as virtual webutil
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded useradd as virtual user
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'port' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'pecl' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'ipvsadm' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded rh_ip as virtual ip
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'mfsgetgoal' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded cmdmod as virtual cmd
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[DEBUG ] LazyLoaded cmd.retcode
[TRACE ] Loaded ini_manage as virtual ini
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded etcd_mod as virtual etcd
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Depends decorator instantiated with dep list of ('zipfile',)
[TRACE ] Depends decorator instantiated with dep list of ('zipfile',)
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded slack_notify as virtual slack
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'monit' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'virtualenv' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'virtualenv-2.7' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'virtualenv-2.6' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'virtualenv2' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded virtualenv_mod as virtual virtualenv
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded mdadm as virtual raid
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'hadoop' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[DEBUG ] Error loading module.ipmi: No module named pyghmi.ipmi
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'riak' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'sysrc' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Loaded linux_acl as virtual acl
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'brew' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'racadm' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'znc' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'nginx' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] 'postfix' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] 'aws' could not be found in the following search path: ['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/bin']
[TRACE ] Loaded linux_lvm as virtual lvm
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Depends decorator instantiated with dep list of ('cx_Oracle',)
[TRACE ] Depends decorator instantiated with dep list of ('cx_Oracle',)
[TRACE ] Depends decorator instantiated with dep list of ('cx_Oracle',)
[TRACE ] Loaded djangomod as virtual django
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function version at 0x7f2cede45b18> because dependency (cx_Oracle) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function client_version at 0x7f2cede45b90> because dependency (cx_Oracle) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function run_query at 0x7f2cede457d0> because dependency (cx_Oracle) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function version at 0x7f2cede45b18> because dependency (cx_Oracle) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function client_version at 0x7f2cede45b90> because dependency (cx_Oracle) is not imported
[TRACE ] Unloading <module 'salt.loaded.int.module.oracle' from '/usr/lib/python2.7/site-packages/salt/modules/oracle.pyc'>.<function run_query at 0x7f2cede457d0> because dependency (cx_Oracle) is not imported
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Dependency (zipfile) already loaded inside archive, skipping
[TRACE ] Unloading <module 'salt.loaded.int.module.test' from '/usr/lib/python2.7/site-packages/salt/modules/test.pyc'>.<function missing_func at 0x7f2cee360758> because dependency (non_existantmodulename) is not imported
[DEBUG ] Could not LazyLoad pkg.ex_mod_init
[INFO ] Running state [vim-enhanced] at time 17:39:24.006911
[INFO ] Executing state pkg.installed for vim-enhanced
[INFO ] Executing command "repoquery --plugins --queryformat '%{NAME}_|-%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-%{REPOID}' --all --pkgnarrow=installed" in directory '/root'
[TRACE ] stdout: GConf2_|-3.2.6_|-15.fc23_|-x86_64_|-installed
GeoIP_|-1.6.6_|-1.fc23_|-x86_64_|-installed
GeoIP-GeoLite-data_|-2015.09_|-1.fc23_|-noarch_|-installed
ImageMagick_|-6.9.1.3_|-0.beta.3.fc23.2_|-x86_64_|-installed
ImageMagick-libs_|-6.9.1.3_|-0.beta.3.fc23.2_|-x86_64_|-installed
LibRaw_|-0.16.2_|-2.fc23_|-x86_64_|-installed
ModemManager_|-1.4.10_|-2.fc23_|-x86_64_|-installed
ModemManager-glib_|-1.4.10_|-2.fc23_|-x86_64_|-installed
NetworkManager_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-adsl_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-bluetooth_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-config-connectivity-fedora_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-glib_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-libnm_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-openconnect_|-1.0.2_|-2.fc23_|-x86_64_|-installed
NetworkManager-openvpn_|-1.0.6_|-3.fc23_|-x86_64_|-installed
NetworkManager-openvpn-gnome_|-1.0.6_|-3.fc23_|-x86_64_|-installed
NetworkManager-pptp_|-1.1.0_|-2.20150428git695d4f2.fc23_|-x86_64_|-installed
NetworkManager-pptp-gnome_|-1.1.0_|-2.20150428git695d4f2.fc23_|-x86_64_|-installed
NetworkManager-team_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-vpnc_|-1.0.6_|-3.fc23_|-x86_64_|-installed
NetworkManager-vpnc-gnome_|-1.0.6_|-3.fc23_|-x86_64_|-installed
NetworkManager-wifi_|-1.0.6_|-8.fc23_|-x86_64_|-installed
NetworkManager-wwan_|-1.0.6_|-8.fc23_|-x86_64_|-installed
ORBit2_|-2.14.19_|-14.fc23_|-x86_64_|-installed
OpenEXR-libs_|-2.2.0_|-4.fc23_|-x86_64_|-installed
PackageKit_|-1.0.10_|-2.fc23_|-x86_64_|-installed
PackageKit-command-not-found_|-1.0.10_|-2.fc23_|-x86_64_|-installed
PackageKit-glib_|-1.0.10_|-2.fc23_|-x86_64_|-installed
PackageKit-gstreamer-plugin_|-1.0.10_|-2.fc23_|-x86_64_|-installed
PackageKit-gtk3-module_|-1.0.10_|-2.fc23_|-x86_64_|-installed
PyYAML_|-3.11_|-9.fc23_|-x86_64_|-installed
SDL_|-1.2.15_|-20.fc23_|-x86_64_|-installed
SDL2_|-2.0.3_|-7.fc23_|-x86_64_|-installed
SDL_image_|-1.2.12_|-12.fc23_|-x86_64_|-installed
aajohan-comfortaa-fonts_|-2.004_|-5.fc23_|-noarch_|-installed
abattis-cantarell-fonts_|-0.0.17.2_|-1.fc23_|-noarch_|-installed
abrt_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-ccpp_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-coredump-helper_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-kerneloops_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-pstoreoops_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-python_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-python3_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-vmcore_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-addon-xorg_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-cli_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-dbus_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-desktop_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-gui_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-gui-libs_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-java-connector_|-1.1.0_|-6.fc23_|-x86_64_|-installed
abrt-libs_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-plugin-bodhi_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-python_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-python3_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-retrace-client_|-2.7.0_|-2.fc23_|-x86_64_|-installed
abrt-tui_|-2.7.0_|-2.fc23_|-x86_64_|-installed
accountsservice_|-0.6.40_|-2.fc23_|-x86_64_|-installed
accountsservice-libs_|-0.6.40_|-2.fc23_|-x86_64_|-installed
acl_|-2.2.52_|-10.fc23_|-x86_64_|-installed
adobe-source-han-sans-cn-fonts_|-1.004_|-1.fc23_|-noarch_|-installed
adobe-source-han-sans-twhk-fonts_|-1.004_|-1.fc22_|-noarch_|-installed
adwaita-cursor-theme_|-3.18.0_|-1.fc23_|-noarch_|-installed
adwaita-gtk2-theme_|-3.18.0_|-1.fc23_|-x86_64_|-installed
adwaita-icon-theme_|-3.18.0_|-1.fc23_|-noarch_|-installed
adwaita-qt-common_|-0.3_|-1.fc23_|-noarch_|-installed
adwaita-qt4_|-0.3_|-1.fc23_|-x86_64_|-installed
aisleriot_|-3.18.1_|-1.fc23_|-x86_64_|-installed
alsa-lib_|-1.0.29_|-2.fc23_|-x86_64_|-installed
alsa-plugins-pulseaudio_|-1.0.29_|-2.fc23_|-x86_64_|-installed
alsa-utils_|-1.0.29_|-2.fc23_|-x86_64_|-installed
anaconda_|-23.19.10_|-1.fc23_|-x86_64_|-installed
anaconda-core_|-23.19.10_|-1.fc23_|-x86_64_|-installed
anaconda-gui_|-23.19.10_|-1.fc23_|-x86_64_|-installed
anaconda-tui_|-23.19.10_|-1.fc23_|-x86_64_|-installed
anaconda-user-help_|-22.4_|-2.fc23_|-noarch_|-installed
anaconda-widgets_|-23.19.10_|-1.fc23_|-x86_64_|-installed
appstream-data_|-23_|-6.fc23_|-noarch_|-installed
apr_|-1.5.2_|-2.fc23_|-x86_64_|-installed
apr-util_|-1.5.4_|-2.fc23_|-x86_64_|-installed
argyllcms_|-1.8.3_|-1.fc23_|-x86_64_|-installed
astyle_|-2.05.1_|-3.fc23_|-x86_64_|-installed
at_|-3.1.16_|-6.fc23_|-x86_64_|-installed
at-spi2-atk_|-2.18.1_|-1.fc23_|-x86_64_|-installed
at-spi2-core_|-2.18.1_|-1.fc23_|-x86_64_|-installed
atk_|-2.18.0_|-1.fc23_|-x86_64_|-installed
atkmm_|-2.24.1_|-1.fc23_|-x86_64_|-installed
atmel-firmware_|-1.3_|-14.fc23_|-noarch_|-installed
atom_|-1.2.0_|-0.1.fc21_|-x86_64_|-installed
attr_|-2.4.47_|-14.fc23_|-x86_64_|-installed
audit_|-2.4.4_|-2.fc23_|-x86_64_|-installed
audit-libs_|-2.4.4_|-2.fc23_|-x86_64_|-installed
audit-libs-python_|-2.4.4_|-2.fc23_|-x86_64_|-installed
audit-libs-python3_|-2.4.4_|-2.fc23_|-x86_64_|-installed
augeas-libs_|-1.4.0_|-2.fc23_|-x86_64_|-installed
authconfig_|-6.2.10_|-10.fc23_|-x86_64_|-installed
autocorr-en_|-5.0.3.2_|-4.fc23_|-noarch_|-installed
autodownloader_|-0.3.0_|-10.fc23_|-noarch_|-installed
autogen-libopts_|-5.18.6_|-1.fc23_|-x86_64_|-installed
avahi_|-0.6.31_|-43.fc23_|-x86_64_|-installed
avahi-autoipd_|-0.6.31_|-43.fc23_|-x86_64_|-installed
avahi-glib_|-0.6.31_|-43.fc23_|-x86_64_|-installed
avahi-gobject_|-0.6.31_|-43.fc23_|-x86_64_|-installed
avahi-libs_|-0.6.31_|-43.fc23_|-x86_64_|-installed
avahi-ui-gtk3_|-0.6.31_|-43.fc23_|-x86_64_|-installed
b43-fwcutter_|-019_|-5.fc23_|-x86_64_|-installed
b43-openfwwf_|-5.2_|-12.fc23_|-noarch_|-installed
baobab_|-3.18.1_|-2.fc23_|-x86_64_|-installed
basesystem_|-11_|-1.fc23_|-noarch_|-installed
bash_|-4.3.42_|-1.fc23_|-x86_64_|-installed
bash-completion_|-2.1_|-8.20150513git1950590.fc23_|-noarch_|-installed
bc_|-1.06.95_|-15.fc23_|-x86_64_|-installed
bcache-tools_|-1.0.8_|-4.fc23_|-x86_64_|-installed
bijiben_|-3.18.1_|-1.fc23_|-x86_64_|-installed
bind-libs_|-9.10.3_|-2.fc23_|-x86_64_|-installed
bind-libs-lite_|-9.10.3_|-2.fc23_|-x86_64_|-installed
bind-license_|-9.10.3_|-2.fc23_|-noarch_|-installed
bind-utils_|-9.10.3_|-2.fc23_|-x86_64_|-installed
bind99-libs_|-9.9.7_|-7.P3.fc23_|-x86_64_|-installed
bind99-license_|-9.9.7_|-7.P3.fc23_|-noarch_|-installed
binutils_|-2.25_|-15.fc23_|-x86_64_|-installed
blivet-data_|-1.12.8_|-1.fc23_|-noarch_|-installed
bluez_|-5.35_|-1.fc23_|-x86_64_|-installed
bluez-cups_|-5.35_|-1.fc23_|-x86_64_|-installed
bluez-libs_|-5.35_|-1.fc23_|-x86_64_|-installed
boost-date-time_|-1.58.0_|-8.fc23_|-x86_64_|-installed
boost-iostreams_|-1.58.0_|-8.fc23_|-x86_64_|-installed
boost-regex_|-1.58.0_|-8.fc23_|-x86_64_|-installed
boost-signals_|-1.58.0_|-8.fc23_|-x86_64_|-installed
boost-system_|-1.58.0_|-8.fc23_|-x86_64_|-installed
boost-thread_|-1.58.0_|-8.fc23_|-x86_64_|-installed
brasero-libs_|-3.12.1_|-2.fc23_|-x86_64_|-installed
bridge-utils_|-1.5_|-12.fc23_|-x86_64_|-installed
brlapi_|-0.6.3_|-10.fc23_|-x86_64_|-installed
brltty_|-5.2_|-10.fc23_|-x86_64_|-installed
btrfs-progs_|-4.2.2_|-1.fc23_|-x86_64_|-installed
bzip2_|-1.0.6_|-17.fc23_|-x86_64_|-installed
bzip2-libs_|-1.0.6_|-17.fc23_|-x86_64_|-installed
c-ares_|-1.10.0_|-5.fc23_|-x86_64_|-installed
ca-certificates_|-2015.2.5_|-1.0.fc23_|-noarch_|-installed
cairo_|-1.14.2_|-2.fc23_|-x86_64_|-installed
cairo-gobject_|-1.14.2_|-2.fc23_|-x86_64_|-installed
cairomm_|-1.12.0_|-1.fc23_|-x86_64_|-installed
caribou_|-0.4.19_|-1.fc23_|-x86_64_|-installed
caribou-gtk2-module_|-0.4.19_|-1.fc23_|-x86_64_|-installed
caribou-gtk3-module_|-0.4.19_|-1.fc23_|-x86_64_|-installed
cdparanoia-libs_|-10.2_|-20.fc23_|-x86_64_|-installed
celt051_|-0.5.1.3_|-10.fc23_|-x86_64_|-installed
check_|-0.10.0_|-1.fc23_|-x86_64_|-installed
checkpolicy_|-2.4_|-1.fc23.1_|-x86_64_|-installed
cheese_|-3.18.1_|-1.fc23_|-x86_64_|-installed
cheese-libs_|-3.18.1_|-1.fc23_|-x86_64_|-installed
chkconfig_|-1.6_|-1.fc23_|-x86_64_|-installed
chrony_|-2.1.1_|-1.fc23_|-x86_64_|-installed
cifs-utils_|-6.4_|-4.fc23_|-x86_64_|-installed
clucene-contribs-lib_|-2.3.3.4_|-21.20130812.e8e3d20git.fc23_|-x86_64_|-installed
clucene-core_|-2.3.3.4_|-21.20130812.e8e3d20git.fc23_|-x86_64_|-installed
clucene09-core_|-0.9.21b_|-15.fc23_|-x86_64_|-installed
clutter_|-1.24.2_|-1.fc23_|-x86_64_|-installed
clutter-gst2_|-2.0.16_|-2.fc23_|-x86_64_|-installed
clutter-gst3_|-3.0.14_|-2.fc23_|-x86_64_|-installed
clutter-gtk_|-1.6.6_|-1.fc23_|-x86_64_|-installed
codeblocks_|-13.12_|-19.fc23_|-x86_64_|-installed
codeblocks-libs_|-13.12_|-19.fc23_|-x86_64_|-installed
cogl_|-1.22.0_|-2.fc23_|-x86_64_|-installed
color-filesystem_|-1_|-16.fc23_|-noarch_|-installed
colord_|-1.2.12_|-1.fc23_|-x86_64_|-installed
colord-gtk_|-0.1.26_|-2.fc23_|-x86_64_|-installed
colord-libs_|-1.2.12_|-1.fc23_|-x86_64_|-installed
comps-extras_|-23_|-3.fc23_|-noarch_|-installed
control-center_|-3.18.1_|-1.fc23_|-x86_64_|-installed
control-center-filesystem_|-3.18.1_|-1.fc23_|-x86_64_|-installed
coreutils_|-8.24_|-4.fc23_|-x86_64_|-installed
corosync_|-2.3.5_|-1.fc23_|-x86_64_|-installed
corosynclib_|-2.3.5_|-1.fc23_|-x86_64_|-installed
cpio_|-2.11_|-36.fc23_|-x86_64_|-installed
cpp_|-5.1.1_|-4.fc23_|-x86_64_|-installed
cracklib_|-2.9.1_|-6.fc23_|-x86_64_|-installed
cracklib-dicts_|-2.9.1_|-6.fc23_|-x86_64_|-installed
crda_|-3.18_2015.10.22_|-1.fc23_|-x86_64_|-installed
createrepo_c_|-0.9.1_|-1.fc23_|-x86_64_|-installed
createrepo_c-libs_|-0.9.1_|-1.fc23_|-x86_64_|-installed
cronie_|-1.5.0_|-3.fc23_|-x86_64_|-installed
cronie-anacron_|-1.5.0_|-3.fc23_|-x86_64_|-installed
crontabs_|-1.11_|-11.20150630git.fc23_|-noarch_|-installed
crypto-policies_|-20151104_|-1.gitf1cba5f.fc23_|-noarch_|-installed
cryptsetup_|-1.6.8_|-2.fc23_|-x86_64_|-installed
cryptsetup-libs_|-1.6.8_|-2.fc23_|-x86_64_|-installed
crystalhd-firmware_|-3.10.0_|-11.fc23_|-noarch_|-installed
cups_|-2.1.0_|-2.fc23_|-x86_64_|-installed
cups-client_|-2.1.0_|-2.fc23_|-x86_64_|-installed
cups-filesystem_|-2.1.0_|-2.fc23_|-noarch_|-installed
cups-filters_|-1.1.0_|-1.fc23_|-x86_64_|-installed
cups-filters-libs_|-1.1.0_|-1.fc23_|-x86_64_|-installed
cups-libs_|-2.1.0_|-2.fc23_|-x86_64_|-installed
cups-pk-helper_|-0.2.5_|-6.fc23_|-x86_64_|-installed
curl_|-7.43.0_|-4.fc23_|-x86_64_|-installed
cyrus-sasl_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
cyrus-sasl-gssapi_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
cyrus-sasl-lib_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
cyrus-sasl-md5_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
cyrus-sasl-plain_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
cyrus-sasl-scram_|-2.1.26_|-25.2.fc23_|-x86_64_|-installed
darkplaces_|-20150825_|-4.fc23_|-x86_64_|-installed
dbus_|-1.10.2_|-1.fc23_|-x86_64_|-installed
dbus-glib_|-0.104_|-3.fc23_|-x86_64_|-installed
dbus-libs_|-1.10.2_|-1.fc23_|-x86_64_|-installed
dbus-python_|-1.2.0_|-8.fc23_|-x86_64_|-installed
dbus-sharp_|-0.7.0_|-14.fc23_|-x86_64_|-installed
dbus-sharp-glib_|-0.5.0_|-12.fc23_|-x86_64_|-installed
dbus-x11_|-1.10.2_|-1.fc23_|-x86_64_|-installed
dconf_|-0.24.0_|-2.fc23_|-x86_64_|-installed
dejavu-fonts-common_|-2.35_|-2.fc23_|-noarch_|-installed
dejavu-sans-fonts_|-2.35_|-2.fc23_|-noarch_|-installed
dejavu-sans-mono-fonts_|-2.35_|-2.fc23_|-noarch_|-installed
dejavu-serif-fonts_|-2.35_|-2.fc23_|-noarch_|-installed
deltarpm_|-3.6_|-11.fc23_|-x86_64_|-installed
desktop-backgrounds-gnome_|-23.0.0_|-1.fc23_|-noarch_|-installed
desktop-file-utils_|-0.22_|-5.fc23_|-x86_64_|-installed
devassistant_|-0.11.2_|-1.fc23_|-noarch_|-installed
devassistant-cli_|-0.11.2_|-1.fc23_|-noarch_|-installed
devassistant-core_|-0.11.2_|-1.fc23_|-noarch_|-installed
devassistant-dap-c_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-common_args_|-0.10.0_|-8.fc23_|-noarch_|-installed
devassistant-dap-cpp_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-custom_|-0.10.0_|-8.fc23_|-noarch_|-installed
devassistant-dap-dap_|-1.3_|-2.fc23_|-noarch_|-installed
devassistant-dap-devassistant_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-docker_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-eclipse_|-0.10.0dev_|-9.fc23_|-noarch_|-installed
devassistant-dap-git_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-github_|-0.10.1_|-8.fc23_|-noarch_|-installed
devassistant-dap-java_|-0.11.1_|-2.fc23_|-noarch_|-installed
devassistant-dap-nodejs_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-perl_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-php_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-python_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-ruby_|-0.11_|-2.fc23_|-noarch_|-installed
devassistant-dap-tito_|-0.10.0dev_|-7.fc23_|-noarch_|-installed
devassistant-dap-vim_|-0.10.0dev_|-7.fc23_|-noarch_|-installed
devassistant-gui_|-0.11.2_|-1.fc23_|-noarch_|-installed
devhelp_|-3.18.1_|-1.fc23_|-x86_64_|-installed
device-mapper_|-1.02.107_|-1.fc23_|-x86_64_|-installed
device-mapper-event_|-1.02.107_|-1.fc23_|-x86_64_|-installed
device-mapper-event-libs_|-1.02.107_|-1.fc23_|-x86_64_|-installed
device-mapper-libs_|-1.02.107_|-1.fc23_|-x86_64_|-installed
device-mapper-multipath_|-0.4.9_|-80.fc23_|-x86_64_|-installed
device-mapper-multipath-libs_|-0.4.9_|-80.fc23_|-x86_64_|-installed
device-mapper-persistent-data_|-0.5.5_|-1.fc23_|-x86_64_|-installed
dhcp-client_|-4.3.3_|-6.fc23_|-x86_64_|-installed
dhcp-common_|-4.3.3_|-6.fc23_|-noarch_|-installed
dhcp-libs_|-4.3.3_|-6.fc23_|-x86_64_|-installed
diffutils_|-3.3_|-12.fc23_|-x86_64_|-installed
djvulibre-libs_|-3.5.25.3_|-17.fc23_|-x86_64_|-installed
dleyna-connector-dbus_|-0.2.0_|-6.fc23_|-x86_64_|-installed
dleyna-core_|-0.5.0_|-2.fc23_|-x86_64_|-installed
dleyna-server_|-0.5.0_|-2.fc23_|-x86_64_|-installed
dmidecode_|-2.12_|-9.fc23_|-x86_64_|-installed
dmraid_|-1.0.0.rc16_|-28.fc23_|-x86_64_|-installed
dmraid-events_|-1.0.0.rc16_|-28.fc23_|-x86_64_|-installed
dnf_|-1.1.3_|-1.fc23_|-noarch_|-installed
dnf-conf_|-1.1.3_|-1.fc23_|-noarch_|-installed
dnf-plugin-system-upgrade_|-0.7.0_|-1.fc23_|-noarch_|-installed
dnf-plugins-core_|-0.1.13_|-1.fc23_|-noarch_|-installed
dnf-yum_|-1.1.3_|-1.fc23_|-noarch_|-installed
dnsmasq_|-2.75_|-2.fc23_|-x86_64_|-installed
dos2unix_|-7.2.3_|-1.fc23_|-x86_64_|-installed
dosfstools_|-3.0.28_|-1.fc23_|-x86_64_|-installed
dotconf_|-1.3_|-11.fc23_|-x86_64_|-installed
dracut_|-043_|-60.git20150811.fc23_|-x86_64_|-installed
dracut-config-rescue_|-043_|-60.git20150811.fc23_|-x86_64_|-installed
dracut-network_|-043_|-60.git20150811.fc23_|-x86_64_|-installed
drpm_|-0.2.0_|-3.fc23_|-x86_64_|-installed
dtc_|-1.4.1_|-4.fc23_|-x86_64_|-installed
e2fsprogs_|-1.42.13_|-3.fc23_|-x86_64_|-installed
e2fsprogs-libs_|-1.42.13_|-3.fc23_|-x86_64_|-installed
ebtables_|-2.0.10_|-17.fc23_|-x86_64_|-installed
ed_|-1.10_|-5.fc23_|-x86_64_|-installed
efibootmgr_|-0.12_|-2.fc23_|-x86_64_|-installed
efivar-libs_|-0.21_|-1.fc23_|-x86_64_|-installed
elfutils_|-0.163_|-4.fc23_|-x86_64_|-installed
elfutils-default-yama-scope_|-0.163_|-4.fc23_|-noarch_|-installed
elfutils-libelf_|-0.163_|-4.fc23_|-x86_64_|-installed
elfutils-libs_|-0.163_|-4.fc23_|-x86_64_|-installed
emacs-filesystem_|-24.5_|-6.fc23_|-noarch_|-installed
empathy_|-3.12.11_|-1.fc23_|-x86_64_|-installed
enca_|-1.16_|-1.fc23_|-x86_64_|-installed
enchant_|-1.6.0_|-12.fc23_|-x86_64_|-installed
environment-modules_|-3.2.10_|-16.fc23_|-x86_64_|-installed
eog_|-3.18.0_|-1.fc23_|-x86_64_|-installed
epiphany-runtime_|-3.18.0_|-4.fc23_|-x86_64_|-installed
esmtp_|-1.0_|-14.fc23_|-x86_64_|-installed
espeak_|-1.48.04_|-6.fc23_|-x86_64_|-installed
ethtool_|-4.0_|-2.fc23_|-x86_64_|-installed
evince_|-3.18.0_|-1.fc23_|-x86_64_|-installed
evince-browser-plugin_|-3.18.0_|-1.fc23_|-x86_64_|-installed
evince-djvu_|-3.18.0_|-1.fc23_|-x86_64_|-installed
evince-libs_|-3.18.0_|-1.fc23_|-x86_64_|-installed
evince-nautilus_|-3.18.0_|-1.fc23_|-x86_64_|-installed
evolution_|-3.18.1_|-1.fc23_|-x86_64_|-installed
evolution-data-server_|-3.18.1_|-1.fc23_|-x86_64_|-installed
evolution-ews_|-3.18.1_|-1.fc23_|-x86_64_|-installed
evolution-help_|-3.18.1_|-1.fc23_|-noarch_|-installed
exempi_|-2.2.1_|-12.fc23_|-x86_64_|-installed
exiv2-libs_|-0.25_|-1.fc23_|-x86_64_|-installed
expat_|-2.1.0_|-12.fc23_|-x86_64_|-installed
f22-backgrounds-base_|-22.1.0_|-1.fc23_|-noarch_|-installed
f22-backgrounds-gnome_|-22.1.0_|-1.fc23_|-noarch_|-installed
f23-backgrounds-base_|-23.1.0_|-1.fc23_|-noarch_|-installed
f23-backgrounds-gnome_|-23.1.0_|-1.fc23_|-noarch_|-installed
farstream02_|-0.2.7_|-2.fc23_|-x86_64_|-installed
fcoe-utils_|-1.0.30_|-3.git91c0c8c.fc23_|-x86_64_|-installed
fedora-bookmarks_|-22_|-2.fc23_|-noarch_|-installed
fedora-logos_|-22.0.0_|-2.fc23_|-x86_64_|-installed
fedora-logos-httpd_|-22.0.0_|-2.fc23_|-noarch_|-installed
fedora-productimg-workstation_|-23_|-1.fc23_|-x86_64_|-installed
fedora-release_|-23_|-1_|-noarch_|-installed
fedora-release-workstation_|-23_|-1_|-noarch_|-installed
fedora-repos_|-23_|-1_|-noarch_|-installed
fedora-user-agent-chrome_|-0.0.0.1_|-2.fc23_|-noarch_|-installed
fftw-libs-double_|-3.3.4_|-6.fc23_|-x86_64_|-installed
file_|-5.22_|-5.fc23_|-x86_64_|-installed
file-libs_|-5.22_|-5.fc23_|-x86_64_|-installed
file-roller_|-3.16.4_|-1.fc23_|-x86_64_|-installed
file-roller-nautilus_|-3.16.4_|-1.fc23_|-x86_64_|-installed
filesystem_|-3.2_|-35.fc23_|-x86_64_|-installed
findutils_|-4.5.14_|-7.fc23_|-x86_64_|-installed
fipscheck_|-1.4.1_|-9.fc23_|-x86_64_|-installed
fipscheck-lib_|-1.4.1_|-9.fc23_|-x86_64_|-installed
firebird-libfbembed_|-2.5.4.26856.0_|-3.fc23_|-x86_64_|-installed
firefox_|-42.0_|-2.fc23_|-x86_64_|-installed
firewalld_|-0.3.14.2_|-4.fc23_|-noarch_|-installed
firewalld-filesystem_|-0.3.14.2_|-4.fc23_|-noarch_|-installed
flac-libs_|-1.3.1_|-5.fc23_|-x86_64_|-installed
folks_|-0.11.1_|-3.fc23_|-x86_64_|-installed
fontconfig_|-2.11.94_|-4.fc23_|-x86_64_|-installed
fontpackages-filesystem_|-1.44_|-14.fc23_|-noarch_|-installed
foomatic_|-4.0.12_|-4.fc23_|-x86_64_|-installed
foomatic-db_|-4.0_|-47.20150819.fc23_|-noarch_|-installed
foomatic-db-filesystem_|-4.0_|-47.20150819.fc23_|-noarch_|-installed
foomatic-db-ppds_|-4.0_|-47.20150819.fc23_|-noarch_|-installed
fpaste_|-0.3.8.1_|-1.fc23_|-noarch_|-installed
fprintd_|-0.6.0_|-3.fc23_|-x86_64_|-installed
fprintd-pam_|-0.6.0_|-3.fc23_|-x86_64_|-installed
freerdp-libs_|-1.2.0_|-0.9.git.24a752a.fc23_|-x86_64_|-installed
freetype_|-2.6.0_|-3.fc23_|-x86_64_|-installed
frei0r-plugins_|-1.4_|-5.fc23_|-x86_64_|-installed
fros_|-1.1_|-6.fc23_|-noarch_|-installed
fros-gnome_|-1.1_|-6.fc23_|-noarch_|-installed
fuse_|-2.9.4_|-3.fc23_|-x86_64_|-installed
fuse-libs_|-2.9.4_|-3.fc23_|-x86_64_|-installed
fwupd_|-0.5.3_|-1.fc23_|-x86_64_|-installed
fwupdate-libs_|-0.4_|-2.fc23_|-x86_64_|-installed
gamin_|-0.1.10_|-18.fc23_|-x86_64_|-installed
gavl_|-1.4.0_|-7.fc23_|-x86_64_|-installed
gawk_|-4.1.3_|-2.fc23_|-x86_64_|-installed
gc_|-7.4.2_|-4.fc23_|-x86_64_|-installed
gcc_|-5.1.1_|-4.fc23_|-x86_64_|-installed
gcr_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gd_|-2.1.1_|-4.fc23_|-x86_64_|-installed
gdb_|-7.10_|-29.fc23_|-x86_64_|-installed
gdbm_|-1.11_|-6.fc23_|-x86_64_|-installed
gdisk_|-1.0.1_|-1.fc23_|-x86_64_|-installed
gdk-pixbuf2_|-2.32.1_|-1.fc23_|-x86_64_|-installed
gdk-pixbuf2-modules_|-2.32.1_|-1.fc23_|-x86_64_|-installed
gdm_|-3.18.0_|-1.fc23_|-x86_64_|-installed
geany_|-1.25_|-6.fc23_|-x86_64_|-installed
geany-libgeany_|-1.25_|-6.fc23_|-x86_64_|-installed
geary_|-0.10.0_|-3.fc23_|-x86_64_|-installed
gedit_|-3.18.1_|-1.fc23_|-x86_64_|-installed
genisoimage_|-1.1.11_|-29.fc23_|-x86_64_|-installed
geoclue2_|-2.2.0_|-2.fc23_|-x86_64_|-installed
geocode-glib_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gettext_|-0.19.6_|-1.fc23_|-x86_64_|-installed
gettext-libs_|-0.19.6_|-1.fc23_|-x86_64_|-installed
gfbgraph_|-0.2.3_|-1.fc23_|-x86_64_|-installed
ghostscript_|-9.16_|-3.fc23_|-x86_64_|-installed
ghostscript-core_|-9.16_|-3.fc23_|-x86_64_|-installed
ghostscript-fonts_|-5.50_|-34.fc23_|-noarch_|-installed
ghostscript-x11_|-9.16_|-3.fc23_|-x86_64_|-installed
giflib_|-4.1.6_|-14.fc23_|-x86_64_|-installed
git_|-2.5.0_|-2.fc23_|-x86_64_|-installed
git-core_|-2.5.0_|-2.fc23_|-x86_64_|-installed
git-core-doc_|-2.5.0_|-2.fc23_|-x86_64_|-installed
gjs_|-1.43.3_|-3.fc23_|-x86_64_|-installed
glade-libs_|-3.19.0_|-3.fc23_|-x86_64_|-installed
glib-networking_|-2.46.1_|-1.fc23_|-x86_64_|-installed
glib2_|-2.46.1_|-2.fc23_|-x86_64_|-installed
glibc_|-2.22_|-5.fc23_|-x86_64_|-installed
glibc-common_|-2.22_|-5.fc23_|-x86_64_|-installed
glibc-devel_|-2.22_|-5.fc23_|-x86_64_|-installed
glibc-headers_|-2.22_|-5.fc23_|-x86_64_|-installed
glibmm24_|-2.46.1_|-1.fc23_|-x86_64_|-installed
glusterfs_|-3.7.5_|-1.fc23_|-x86_64_|-installed
glusterfs-api_|-3.7.5_|-1.fc23_|-x86_64_|-installed
glusterfs-client-xlators_|-3.7.5_|-1.fc23_|-x86_64_|-installed
glusterfs-fuse_|-3.7.5_|-1.fc23_|-x86_64_|-installed
glusterfs-libs_|-3.7.5_|-1.fc23_|-x86_64_|-installed
glx-utils_|-8.2.0_|-4.fc23_|-x86_64_|-installed
gmime_|-2.6.20_|-5.fc23_|-x86_64_|-installed
gmp_|-6.0.0_|-12.fc23_|-x86_64_|-installed
gnome-abrt_|-1.2.0_|-5.fc23_|-x86_64_|-installed
gnome-backgrounds_|-3.18.0_|-1.fc23_|-noarch_|-installed
gnome-bluetooth_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-bluetooth-libs_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-calculator_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-classic-session_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-clocks_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-color-manager_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-contacts_|-3.18.0_|-2.fc23_|-x86_64_|-installed
gnome-desktop3_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-disk-utility_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-documents_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-documents-libs_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-font-viewer_|-3.16.2_|-3.fc23_|-x86_64_|-installed
gnome-getting-started-docs_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-initial-setup_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-keyring_|-3.18.1_|-2.fc23_|-x86_64_|-installed
gnome-keyring-pam_|-3.18.1_|-2.fc23_|-x86_64_|-installed
gnome-maps_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-menus_|-3.13.3_|-4.fc23_|-x86_64_|-installed
gnome-online-accounts_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-online-miners_|-3.14.3_|-3.fc23_|-x86_64_|-installed
gnome-power-manager_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-screenshot_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-session_|-3.18.1.2_|-2.fc23_|-x86_64_|-installed
gnome-session-wayland-session_|-3.18.1.2_|-2.fc23_|-x86_64_|-installed
gnome-session-xsession_|-3.18.1.2_|-2.fc23_|-x86_64_|-installed
gnome-settings-daemon_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-shell_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-shell-extension-alternate-tab_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-apps-menu_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-background-logo_|-3.18.0_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-common_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-launch-new-instance_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-places-menu_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-user-theme_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-shell-extension-window-list_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-software_|-3.18.3_|-1.fc23_|-x86_64_|-installed
gnome-system-log_|-3.9.90_|-5.fc23_|-x86_64_|-installed
gnome-system-monitor_|-3.18.0.1_|-1.fc23_|-x86_64_|-installed
gnome-terminal_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gnome-themes-standard_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-tweak-tool_|-3.18.0_|-1.fc23_|-noarch_|-installed
gnome-user-docs_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnome-user-share_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gnome-vfs2_|-2.24.4_|-19.fc23_|-x86_64_|-installed
gnome-vfs2-common_|-2.24.4_|-19.fc23_|-noarch_|-installed
gnome-video-effects_|-0.4.1_|-3.fc23_|-noarch_|-installed
gnome-weather_|-3.18.1_|-1.fc23_|-noarch_|-installed
gnu-free-fonts-common_|-20120503_|-11.fc23_|-noarch_|-installed
gnu-free-mono-fonts_|-20120503_|-11.fc23_|-noarch_|-installed
gnu-free-sans-fonts_|-20120503_|-11.fc23_|-noarch_|-installed
gnu-free-serif-fonts_|-20120503_|-11.fc23_|-noarch_|-installed
gnupg_|-1.4.19_|-3.fc23_|-x86_64_|-installed
gnupg2_|-2.1.9_|-1.fc23_|-x86_64_|-installed
gnutls_|-3.4.6_|-1.fc23_|-x86_64_|-installed
gnutls-dane_|-3.4.6_|-1.fc23_|-x86_64_|-installed
gnutls-utils_|-3.4.6_|-1.fc23_|-x86_64_|-installed
go-srpm-macros_|-2_|-3.fc23_|-noarch_|-installed
gobject-introspection_|-1.46.0_|-1.fc23_|-x86_64_|-installed
golang_|-1.5.1_|-0.fc23_|-x86_64_|-installed
golang-bin_|-1.5.1_|-0.fc23_|-x86_64_|-installed
golang-src_|-1.5.1_|-0.fc23_|-noarch_|-installed
gom_|-0.3.1_|-2.fc23_|-x86_64_|-installed
google-android-emoji-fonts_|-1.01_|-0.4.20120228git.fc23_|-noarch_|-installed
google-crosextra-caladea-fonts_|-1.002_|-0.6.20130214.fc23_|-noarch_|-installed
google-crosextra-carlito-fonts_|-1.103_|-0.4.20130920.fc23_|-noarch_|-installed
google-noto-fonts-common_|-20150417_|-2.fc23_|-noarch_|-installed
google-noto-sans-lisu-fonts_|-20150417_|-2.fc23_|-noarch_|-installed
google-noto-sans-mandaic-fonts_|-20150417_|-2.fc23_|-noarch_|-installed
google-noto-sans-tagalog-fonts_|-20150417_|-2.fc23_|-noarch_|-installed
google-noto-sans-tai-tham-fonts_|-20150417_|-2.fc23_|-noarch_|-installed
google-noto-sans-tai-viet-fonts_|-20150417_|-2.fc23_|-noarch_|-installed
gparted_|-0.23.0_|-3.fc23_|-x86_64_|-installed
gperftools-libs_|-2.4_|-5.fc23_|-x86_64_|-installed
gpgme_|-1.4.3_|-6.fc23_|-x86_64_|-installed
gpm-libs_|-1.20.7_|-7.fc23_|-x86_64_|-installed
graphite2_|-1.2.4_|-5.fc23_|-x86_64_|-installed
grep_|-2.22_|-2.fc23_|-x86_64_|-installed
grilo_|-0.2.15_|-1.fc23_|-x86_64_|-installed
grilo-plugins_|-0.2.16_|-1.fc23_|-x86_64_|-installed
groff-base_|-1.22.3_|-6.fc23_|-x86_64_|-installed
grub2_|-2.02_|-0.23.fc23_|-x86_64_|-installed
grub2-efi_|-2.02_|-0.23.fc23_|-x86_64_|-installed
grub2-tools_|-2.02_|-0.23.fc23_|-x86_64_|-installed
grubby_|-8.40_|-2.fc23_|-x86_64_|-installed
gsettings-desktop-schemas_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gsm_|-1.0.13_|-13.fc23_|-x86_64_|-installed
gsound_|-1.0.1_|-2.fc23_|-x86_64_|-installed
gssdp_|-0.14.11_|-2.fc23_|-x86_64_|-installed
gssproxy_|-0.4.1_|-3.fc23_|-x86_64_|-installed
gstreamer-plugin-crystalhd_|-3.10.0_|-11.fc23_|-x86_64_|-installed
gstreamer1_|-1.6.1_|-1.fc23_|-x86_64_|-installed
gstreamer1-plugins-bad-free_|-1.6.1_|-2.fc23_|-x86_64_|-installed
gstreamer1-plugins-base_|-1.6.1_|-1.fc23_|-x86_64_|-installed
gstreamer1-plugins-good_|-1.6.1_|-1.fc23_|-x86_64_|-installed
gtk-sharp2_|-2.12.26_|-2.fc23_|-x86_64_|-installed
gtk-update-icon-cache_|-3.18.2_|-1.fc23_|-x86_64_|-installed
gtk-vnc2_|-0.5.4_|-2.fc23_|-x86_64_|-installed
gtk2_|-2.24.28_|-2.fc23_|-x86_64_|-installed
gtk3_|-3.18.2_|-1.fc23_|-x86_64_|-installed
gtkmm24_|-2.24.4_|-7.fc23_|-x86_64_|-installed
gtkmm30_|-3.18.0_|-1.fc23_|-x86_64_|-installed
gtksourceview3_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gtkspell3_|-3.0.7_|-2.fc23_|-x86_64_|-installed
gucharmap_|-3.18.1_|-1.fc23_|-x86_64_|-installed
gucharmap-libs_|-3.18.1_|-1.fc23_|-x86_64_|-installed
guile_|-2.0.11_|-6.fc23_|-x86_64_|-installed
gupnp_|-0.20.14_|-3.fc23_|-x86_64_|-installed
gupnp-av_|-0.12.7_|-2.fc23_|-x86_64_|-installed
gupnp-dlna_|-0.10.3_|-1.fc23_|-x86_64_|-installed
gupnp-igd_|-0.2.3_|-5.fc23_|-x86_64_|-installed
gutenprint_|-5.2.10_|-15.fc23_|-x86_64_|-installed
gutenprint-cups_|-5.2.10_|-15.fc23_|-x86_64_|-installed
gutenprint-libs_|-5.2.10_|-15.fc23_|-x86_64_|-installed
gvfs_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-afc_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-afp_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-archive_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-client_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-fuse_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-goa_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-gphoto2_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-mtp_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvfs-smb_|-1.26.1.1_|-2.fc23_|-x86_64_|-installed
gvnc_|-0.5.4_|-2.fc23_|-x86_64_|-installed
gzip_|-1.6_|-9.fc23_|-x86_64_|-installed
hardlink_|-1.0_|-22.fc23_|-x86_64_|-installed
harfbuzz_|-1.0.6_|-1.fc23_|-x86_64_|-installed
harfbuzz-icu_|-1.0.6_|-1.fc23_|-x86_64_|-installed
hawkey_|-0.6.2_|-1.fc23_|-x86_64_|-installed
hfsplus-tools_|-540.1.linux3_|-10.fc23_|-x86_64_|-installed
hicolor-icon-theme_|-0.15_|-2.fc23_|-noarch_|-installed
highlight_|-3.22_|-4.fc23_|-x86_64_|-installed
hostname_|-3.15_|-6.fc23_|-x86_64_|-installed
hplip_|-3.15.9_|-2.fc23_|-x86_64_|-installed
hplip-common_|-3.15.9_|-2.fc23_|-x86_64_|-installed
hplip-libs_|-3.15.9_|-2.fc23_|-x86_64_|-installed
httpd_|-2.4.17_|-3.fc23_|-x86_64_|-installed
httpd-filesystem_|-2.4.17_|-3.fc23_|-noarch_|-installed
httpd-tools_|-2.4.17_|-3.fc23_|-x86_64_|-installed
hunspell_|-1.3.3_|-8.fc23_|-x86_64_|-installed
hunspell-en-US_|-0.20140811.1_|-2.fc23_|-noarch_|-installed
hwdata_|-0.283_|-1.fc23_|-noarch_|-installed
hyperv-daemons_|-0_|-0.12.20150702git.fc23_|-x86_64_|-installed
hyperv-daemons-license_|-0_|-0.12.20150702git.fc23_|-noarch_|-installed
hypervfcopyd_|-0_|-0.12.20150702git.fc23_|-x86_64_|-installed
hypervkvpd_|-0_|-0.12.20150702git.fc23_|-x86_64_|-installed
hypervvssd_|-0_|-0.12.20150702git.fc23_|-x86_64_|-installed
hyphen_|-2.8.8_|-2.fc23_|-x86_64_|-installed
hyphen-en_|-2.8.8_|-2.fc23_|-noarch_|-installed
ibus_|-1.5.11_|-1.fc23_|-x86_64_|-installed
ibus-chewing_|-1.4.14_|-4.fc23_|-x86_64_|-installed
ibus-gtk2_|-1.5.11_|-1.fc23_|-x86_64_|-installed
ibus-gtk3_|-1.5.11_|-1.fc23_|-x86_64_|-installed
ibus-hangul_|-1.5.0_|-5.fc23_|-x86_64_|-installed
ibus-kkc_|-1.5.22_|-3.fc23_|-x86_64_|-installed
ibus-libpinyin_|-1.7.2_|-1.fc23_|-x86_64_|-installed
ibus-libs_|-1.5.11_|-1.fc23_|-x86_64_|-installed
ibus-m17n_|-1.3.4_|-19.fc23_|-x86_64_|-installed
ibus-qt_|-1.3.3_|-8.fc23_|-x86_64_|-installed
ibus-rawcode_|-1.3.2_|-6.fc23_|-x86_64_|-installed
ibus-setup_|-1.5.11_|-1.fc23_|-noarch_|-installed
ibus-typing-booster_|-1.2.14_|-2.fc23_|-noarch_|-installed
ibus-wayland_|-1.5.11_|-1.fc23_|-x86_64_|-installed
iio-sensor-proxy_|-1.1_|-1.fc23_|-x86_64_|-installed
ilmbase_|-2.2.0_|-4.fc23_|-x86_64_|-installed
info_|-6.0_|-1.fc23_|-x86_64_|-installed
initscripts_|-9.64_|-1.fc23_|-x86_64_|-installed
ipcalc_|-0.1.4_|-2.fc23_|-x86_64_|-installed
iproute_|-4.1.1_|-3.fc23_|-x86_64_|-installed
iptables_|-1.4.21_|-15.fc23_|-x86_64_|-installed
iptstate_|-2.2.5_|-7.fc23_|-x86_64_|-installed
iputils_|-20140519_|-6.fc23_|-x86_64_|-installed
ipw2100-firmware_|-1.3_|-18.fc23_|-noarch_|-installed
ipw2200-firmware_|-3.1_|-11.fc23_|-noarch_|-installed
ipxe-roms-qemu_|-20150407_|-3.gitdc795b9f.fc23_|-noarch_|-installed
irssi_|-0.8.17_|-4.fc23_|-x86_64_|-installed
iscsi-initiator-utils_|-6.2.0.873_|-29.git4c9d6f9.fc23_|-x86_64_|-installed
iscsi-initiator-utils-iscsiuio_|-6.2.0.873_|-29.git4c9d6f9.fc23_|-x86_64_|-installed
isl_|-0.14_|-4.fc23_|-x86_64_|-installed
iso-codes_|-3.62_|-1.fc23_|-noarch_|-installed
isomd5sum_|-1.0.12_|-6.fc23_|-x86_64_|-installed
iw_|-4.1_|-1.fc23_|-x86_64_|-installed
iwl100-firmware_|-39.31.5.1_|-58.fc23_|-noarch_|-installed
iwl1000-firmware_|-39.31.5.1_|-58.fc23_|-noarch_|-installed
iwl105-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl135-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl2000-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl2030-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl3160-firmware_|-25.30.13.0_|-58.fc23_|-noarch_|-installed
iwl3945-firmware_|-15.32.2.9_|-58.fc23_|-noarch_|-installed
iwl4965-firmware_|-228.61.2.24_|-58.fc23_|-noarch_|-installed
iwl5000-firmware_|-8.83.5.1_1_|-58.fc23_|-noarch_|-installed
iwl5150-firmware_|-8.24.2.2_|-58.fc23_|-noarch_|-installed
iwl6000-firmware_|-9.221.4.1_|-58.fc23_|-noarch_|-installed
iwl6000g2a-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl6000g2b-firmware_|-18.168.6.1_|-58.fc23_|-noarch_|-installed
iwl6050-firmware_|-41.28.5.1_|-58.fc23_|-noarch_|-installed
iwl7260-firmware_|-25.30.13.0_|-58.fc23_|-noarch_|-installed
jack-audio-connection-kit_|-1.9.10_|-3.fc23_|-x86_64_|-installed
jack-audio-connection-kit-example-clients_|-1.9.10_|-3.fc23_|-x86_64_|-installed
jansson_|-2.7_|-2.fc23_|-x86_64_|-installed
jasper-libs_|-1.900.1_|-31.fc23_|-x86_64_|-installed
java-1.8.0-openjdk-headless_|-1.8.0.65_|-3.b17.fc23_|-x86_64_|-installed
javapackages-tools_|-4.6.0_|-7.fc23_|-noarch_|-installed
jbigkit-libs_|-2.1_|-4.fc23_|-x86_64_|-installed
jimtcl_|-0.76_|-3.fc23_|-x86_64_|-installed
jomolhari-fonts_|-0.003_|-20.fc23_|-noarch_|-installed
js-jquery_|-2.1.3_|-2.fc23_|-noarch_|-installed
json-c_|-0.12_|-6.fc23_|-x86_64_|-installed
json-glib_|-1.0.4_|-2.fc23_|-x86_64_|-installed
jwhois_|-4.0_|-43.fc23_|-x86_64_|-installed
kbd_|-2.0.2_|-7.fc23_|-x86_64_|-installed
kbd-legacy_|-2.0.2_|-7.fc23_|-noarch_|-installed
kbd-misc_|-2.0.2_|-7.fc23_|-noarch_|-installed
kernel_|-4.0.4_|-303.fc22_|-x86_64_|-installed
kernel_|-4.2.5_|-201.fc22_|-x86_64_|-installed
kernel_|-4.2.5_|-300.fc23_|-x86_64_|-installed
kernel-core_|-4.0.4_|-303.fc22_|-x86_64_|-installed
kernel-core_|-4.2.5_|-201.fc22_|-x86_64_|-installed
kernel-core_|-4.2.5_|-300.fc23_|-x86_64_|-installed
kernel-headers_|-4.2.5_|-300.fc23_|-x86_64_|-installed
kernel-modules_|-4.0.4_|-303.fc22_|-x86_64_|-installed
kernel-modules_|-4.2.5_|-201.fc22_|-x86_64_|-installed
kernel-modules_|-4.2.5_|-300.fc23_|-x86_64_|-installed
kexec-tools_|-2.0.10_|-4.fc23_|-x86_64_|-installed
keybinder3_|-0.3.0_|-5.fc23_|-x86_64_|-installed
keyutils_|-1.5.9_|-7.fc23_|-x86_64_|-installed
keyutils-libs_|-1.5.9_|-7.fc23_|-x86_64_|-installed
keyutils-libs-devel_|-1.5.9_|-7.fc23_|-x86_64_|-installed
khmeros-base-fonts_|-5.0_|-20.fc23_|-noarch_|-installed
khmeros-fonts-common_|-5.0_|-20.fc23_|-noarch_|-installed
kmod_|-21_|-3.fc23_|-x86_64_|-installed
kmod-libs_|-21_|-3.fc23_|-x86_64_|-installed
kpartx_|-0.4.9_|-80.fc23_|-x86_64_|-installed
krb5-devel_|-1.13.2_|-13.fc23_|-x86_64_|-installed
krb5-libs_|-1.13.2_|-13.fc23_|-x86_64_|-installed
langtable_|-0.0.34_|-1.fc23_|-noarch_|-installed
langtable-data_|-0.0.34_|-1.fc23_|-noarch_|-installed
langtable-python_|-0.0.34_|-1.fc23_|-noarch_|-installed
langtable-python3_|-0.0.34_|-1.fc23_|-noarch_|-installed
latexila_|-3.17.0_|-1.fc23_|-x86_64_|-installed
latexmk_|-4.42_|-2.fc23_|-noarch_|-installed
lcms2_|-2.7_|-2.fc23_|-x86_64_|-installed
less_|-481_|-1.fc23_|-x86_64_|-installed
libGLEW_|-1.10.0_|-6.fc23_|-x86_64_|-installed
libICE_|-1.0.9_|-3.fc23_|-x86_64_|-installed
libIDL_|-0.8.14_|-10.fc23_|-x86_64_|-installed
libSM_|-1.2.2_|-3.fc23_|-x86_64_|-installed
libX11_|-1.6.3_|-2.fc23_|-x86_64_|-installed
libX11-common_|-1.6.3_|-2.fc23_|-noarch_|-installed
libXScrnSaver_|-1.2.2_|-9.fc23_|-x86_64_|-installed
libXau_|-1.0.8_|-5.fc23_|-x86_64_|-installed
libXaw_|-1.0.13_|-2.fc23_|-x86_64_|-installed
libXcomposite_|-0.4.4_|-7.fc23_|-x86_64_|-installed
libXcursor_|-1.1.14_|-4.fc23_|-x86_64_|-installed
libXdamage_|-1.1.4_|-7.fc23_|-x86_64_|-installed
libXdmcp_|-1.1.2_|-2.fc23_|-x86_64_|-installed
libXevie_|-1.0.3_|-10.fc23_|-x86_64_|-installed
libXext_|-1.3.3_|-3.fc23_|-x86_64_|-installed
libXfixes_|-5.0.1_|-5.fc23_|-x86_64_|-installed
libXfont_|-1.5.1_|-3.fc23_|-x86_64_|-installed
libXft_|-2.3.2_|-3.fc23_|-x86_64_|-installed
libXi_|-1.7.4_|-3.fc23_|-x86_64_|-installed
libXinerama_|-1.1.3_|-5.fc23_|-x86_64_|-installed
libXmu_|-1.1.2_|-3.fc23_|-x86_64_|-installed
libXpm_|-3.5.11_|-4.fc23_|-x86_64_|-installed
libXrandr_|-1.5.0_|-2.fc23_|-x86_64_|-installed
libXrender_|-0.9.9_|-2.fc23_|-x86_64_|-installed
libXres_|-1.0.7_|-5.fc23_|-x86_64_|-installed
libXt_|-1.1.5_|-2.fc23_|-x86_64_|-installed
libXtst_|-1.2.2_|-5.fc23_|-x86_64_|-installed
libXv_|-1.0.10_|-3.fc23_|-x86_64_|-installed
libXvMC_|-1.0.9_|-2.fc23_|-x86_64_|-installed
libXxf86dga_|-1.1.4_|-5.fc23_|-x86_64_|-installed
libXxf86misc_|-1.0.3_|-10.fc23_|-x86_64_|-installed
libXxf86vm_|-1.1.4_|-2.fc23_|-x86_64_|-installed
libabw_|-0.1.1_|-7.fc23_|-x86_64_|-installed
libacl_|-2.2.52_|-10.fc23_|-x86_64_|-installed
libaio_|-0.3.110_|-5.fc23_|-x86_64_|-installed
libao_|-1.2.0_|-5.fc23_|-x86_64_|-installed
libappstream-glib_|-0.5.3_|-1.fc23_|-x86_64_|-installed
libarchive_|-3.1.2_|-13.fc23_|-x86_64_|-installed
libassuan_|-2.3.0_|-1.fc23_|-x86_64_|-installed
libasyncns_|-0.8_|-9.fc23_|-x86_64_|-installed
libatasmart_|-0.19_|-8.fc23_|-x86_64_|-installed
libatomic_ops_|-7.4.2_|-6.fc23_|-x86_64_|-installed
libattr_|-2.4.47_|-14.fc23_|-x86_64_|-installed
libavc1394_|-0.5.3_|-18.fc23_|-x86_64_|-installed
libbabeltrace_|-1.2.4_|-2.fc23_|-x86_64_|-installed
libbasicobjects_|-0.1.1_|-27.fc23_|-x86_64_|-installed
libblkid_|-2.27.1_|-1.fc23_|-x86_64_|-installed
libblockdev_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-btrfs_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-crypto_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-dm_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-kbd_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-loop_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-lvm_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-mdraid_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-mpath_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-plugins-all_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-swap_|-1.1_|-2.fc23_|-x86_64_|-installed
libblockdev-utils_|-1.1_|-2.fc23_|-x86_64_|-installed
libbluray_|-0.8.0_|-3.fc23_|-x86_64_|-installed
libbonobo_|-2.32.1_|-8.fc22_|-x86_64_|-installed
libcacard_|-2.4.1_|-1.fc23_|-x86_64_|-installed
libcanberra_|-0.30_|-10.fc23_|-x86_64_|-installed
libcanberra-gtk2_|-0.30_|-10.fc23_|-x86_64_|-installed
libcanberra-gtk3_|-0.30_|-10.fc23_|-x86_64_|-installed
libcap_|-2.24_|-8.fc23_|-x86_64_|-installed
libcap-ng_|-0.7.7_|-2.fc23_|-x86_64_|-installed
libcdio_|-0.93_|-6.fc23_|-x86_64_|-installed
libcdio-paranoia_|-10.2+0.93+1_|-4.fc23_|-x86_64_|-installed
libcdr_|-0.1.1_|-6.fc23_|-x86_64_|-installed
libcgroup_|-0.41_|-7.fc23_|-x86_64_|-installed
libchamplain_|-0.12.11_|-1.fc23_|-x86_64_|-installed
libchamplain-gtk_|-0.12.11_|-1.fc23_|-x86_64_|-installed
libchewing_|-0.4.0_|-5.fc23_|-x86_64_|-installed
libcmis_|-0.5.0_|-8.fc23_|-x86_64_|-installed
libcollection_|-0.7.0_|-27.fc23_|-x86_64_|-installed
libcom_err_|-1.42.13_|-3.fc23_|-x86_64_|-installed
libcom_err-devel_|-1.42.13_|-3.fc23_|-x86_64_|-installed
libcomps_|-0.1.7_|-1.fc23_|-x86_64_|-installed
libconfig_|-1.5_|-2.fc23_|-x86_64_|-installed
libcroco_|-0.6.8_|-7.fc23_|-x86_64_|-installed
libcryptui_|-3.12.2_|-6.fc23_|-x86_64_|-installed
libcrystalhd_|-3.10.0_|-11.fc23_|-x86_64_|-installed
libcue_|-1.4.0_|-6.fc23_|-x86_64_|-installed
libcurl_|-7.43.0_|-4.fc23_|-x86_64_|-installed
libcurl-devel_|-7.43.0_|-4.fc23_|-x86_64_|-installed
libdaemon_|-0.14_|-9.fc23_|-x86_64_|-installed
libdatrie_|-0.2.8_|-6.fc23_|-x86_64_|-installed
libdb_|-5.3.28_|-13.fc23_|-x86_64_|-installed
libdb-utils_|-5.3.28_|-13.fc23_|-x86_64_|-installed
libdhash_|-0.4.3_|-27.fc23_|-x86_64_|-installed
libdmapsharing_|-2.9.30_|-3.fc23_|-x86_64_|-installed
libdmx_|-1.1.3_|-5.fc23_|-x86_64_|-installed
libdnet_|-1.12_|-16.fc23_|-x86_64_|-installed
libdrm_|-2.4.65_|-1.fc23_|-x86_64_|-installed
libdv_|-1.0.0_|-20.fc23_|-x86_64_|-installed
libdvdnav_|-5.0.3_|-2.fc23_|-x86_64_|-installed
libdvdread_|-5.0.2_|-2.fc23_|-x86_64_|-installed
libe-book_|-0.1.2_|-6.fc23_|-x86_64_|-installed
libedit_|-3.1_|-13.20150325cvs.fc23_|-x86_64_|-installed
libeot_|-0.01_|-4.fc23_|-x86_64_|-installed
libepoxy_|-1.2_|-4.fc23_|-x86_64_|-installed
liberation-fonts-common_|-1.07.4_|-6.fc23_|-noarch_|-installed
liberation-mono-fonts_|-1.07.4_|-6.fc23_|-noarch_|-installed
liberation-sans-fonts_|-1.07.4_|-6.fc23_|-noarch_|-installed
liberation-serif-fonts_|-1.07.4_|-6.fc23_|-noarch_|-installed
libertas-usb8388-firmware_|-20151030_|-58.git66d3d8d7.fc23_|-noarch_|-installed
libesmtp_|-1.0.6_|-9.fc23_|-x86_64_|-installed
libetonyek_|-0.1.3_|-3.fc23_|-x86_64_|-installed
libevdev_|-1.4.4_|-1.fc23_|-x86_64_|-installed
libevent_|-2.0.21_|-7.fc23_|-x86_64_|-installed
libexif_|-0.6.21_|-9.fc23_|-x86_64_|-installed
libexttextcat_|-3.4.4_|-3.fc23_|-x86_64_|-installed
libfdisk_|-2.27.1_|-1.fc23_|-x86_64_|-installed
libfdt_|-1.4.1_|-4.fc23_|-x86_64_|-installed
libffado_|-2.2.1_|-5.fc23_|-x86_64_|-installed
libffi_|-3.1_|-8.fc23_|-x86_64_|-installed
libfontenc_|-1.1.3_|-2.fc23_|-x86_64_|-installed
libfprint_|-0.6.0_|-2.fc23_|-x86_64_|-installed
libfreehand_|-0.1.1_|-2.fc23_|-x86_64_|-installed
libgadu_|-1.12.1_|-2.fc23_|-x86_64_|-installed
libgcab1_|-0.6_|-3.fc23_|-x86_64_|-installed
libgcc_|-5.1.1_|-4.fc23_|-x86_64_|-installed
libgcrypt_|-1.6.4_|-1.fc23_|-x86_64_|-installed
libgdata_|-0.17.3_|-1.fc23_|-x86_64_|-installed
libgdiplus_|-3.12_|-3.fc23_|-x86_64_|-installed
libgdither_|-0.6_|-10.fc23_|-x86_64_|-installed
libgee_|-0.18.0_|-2.fc23_|-x86_64_|-installed
libgexiv2_|-0.10.3_|-4.fc23_|-x86_64_|-installed
libglade2_|-2.6.4_|-13.fc23_|-x86_64_|-installed
libgltf_|-0.0.2_|-8.fc23_|-x86_64_|-installed
libgnome_|-2.32.1_|-10.fc23_|-x86_64_|-installed
libgnome-keyring_|-3.12.0_|-5.fc23_|-x86_64_|-installed
libgnomekbd_|-3.6.0_|-8.fc23_|-x86_64_|-installed
libgomp_|-5.1.1_|-4.fc23_|-x86_64_|-installed
libgovirt_|-0.3.3_|-2.fc23_|-x86_64_|-installed
libgpg-error_|-1.20_|-1.fc23_|-x86_64_|-installed
libgphoto2_|-2.5.7_|-4.fc23_|-x86_64_|-installed
libgpod_|-0.8.3_|-10.fc23_|-x86_64_|-installed
libgrss_|-0.7.0_|-1.fc23_|-x86_64_|-installed
libgsf_|-1.14.33_|-2.fc23_|-x86_64_|-installed
libgsystem_|-2015.2_|-1.fc23_|-x86_64_|-installed
libgtop2_|-2.32.0_|-1.fc23_|-x86_64_|-installed
libgudev_|-230_|-2.fc23_|-x86_64_|-installed
libgusb_|-0.2.7_|-1.fc23_|-x86_64_|-installed
libgweather_|-3.18.1_|-1.fc23_|-x86_64_|-installed
libgxps_|-0.2.3.2_|-1.fc23_|-x86_64_|-installed
libhangul_|-0.1.0_|-11.fc23_|-x86_64_|-installed
libhbaapi_|-2.2.9_|-7.fc23_|-x86_64_|-installed
libhbalinux_|-1.0.17_|-2.fc23_|-x86_64_|-installed
libhif_|-0.2.1_|-4.fc23_|-x86_64_|-installed
libibverbs_|-1.1.8_|-4.fc23_|-x86_64_|-installed
libical_|-1.0.1_|-2.fc23_|-x86_64_|-installed
libicu_|-54.1_|-5.fc23_|-x86_64_|-installed
libidn_|-1.32_|-1.fc23_|-x86_64_|-installed
libidn2_|-0.10_|-3.fc23_|-x86_64_|-installed
libiec61883_|-1.2.0_|-13.fc23_|-x86_64_|-installed
libieee1284_|-0.2.11_|-18.fc23_|-x86_64_|-installed
libimobiledevice_|-1.2.0_|-3.fc23_|-x86_64_|-installed
libini_config_|-1.2.0_|-27.fc23_|-x86_64_|-installed
libinput_|-1.1.0_|-3.fc23_|-x86_64_|-installed
libipa_hbac_|-1.13.1_|-3.fc23_|-x86_64_|-installed
libipt_|-1.4.3_|-1.fc23_|-x86_64_|-installed
libiptcdata_|-1.0.4_|-13.fc23_|-x86_64_|-installed
libiscsi_|-1.15.0_|-1.fc23_|-x86_64_|-installed
libjpeg-turbo_|-1.4.1_|-2.fc23_|-x86_64_|-installed
libkkc_|-0.3.5_|-3.fc23_|-x86_64_|-installed
libkkc-common_|-0.3.5_|-3.fc23_|-noarch_|-installed
libkkc-data_|-0.2.7_|-5.fc23_|-x86_64_|-installed
libksba_|-1.3.3_|-2.fc23_|-x86_64_|-installed
liblangtag_|-0.5.7_|-1.fc23_|-x86_64_|-installed
libldb_|-1.1.21_|-1.fc23_|-x86_64_|-installed
liblouis_|-2.6.2_|-3.fc23_|-x86_64_|-installed
libmbim_|-1.12.2_|-1.fc23_|-x86_64_|-installed
libmbim-utils_|-1.12.2_|-1.fc23_|-x86_64_|-installed
libmcpp_|-2.7.2_|-15.fc23_|-x86_64_|-installed
libmediaart_|-1.9.0_|-3.fc23_|-x86_64_|-installed
libmetalink_|-0.1.2_|-8.fc23_|-x86_64_|-installed
libmikmod_|-3.3.7_|-2.fc23_|-x86_64_|-installed
libmng_|-2.0.3_|-2.fc23_|-x86_64_|-installed
libmnl_|-1.0.3_|-10.fc23_|-x86_64_|-installed
libmodman_|-2.0.1_|-11.fc23_|-x86_64_|-installed
libmount_|-2.27.1_|-1.fc23_|-x86_64_|-installed
libmpc_|-1.0.2_|-4.fc23_|-x86_64_|-installed
libmspack_|-0.5_|-0.2.alpha.fc23_|-x86_64_|-installed
libmspub_|-0.1.2_|-6.fc23_|-x86_64_|-installed
libmtp_|-1.1.9_|-2.fc23_|-x86_64_|-installed
libmusicbrainz5_|-5.1.0_|-4.fc23_|-x86_64_|-installed
libmwaw_|-0.3.6_|-1.fc23_|-x86_64_|-installed
libmx_|-1.4.7_|-16.fc23_|-x86_64_|-installed
libndp_|-1.5_|-2.fc23_|-x86_64_|-installed
libnetfilter_conntrack_|-1.0.4_|-5.fc23_|-x86_64_|-installed
libnfnetlink_|-1.0.1_|-7.fc23_|-x86_64_|-installed
libnfsidmap_|-0.26_|-3.1.fc23_|-x86_64_|-installed
libnghttp2_|-1.3.3_|-1.fc23_|-x86_64_|-installed
libnice_|-0.1.13_|-2.fc23_|-x86_64_|-installed
libnice-gstreamer1_|-0.1.13_|-2.fc23_|-x86_64_|-installed
libnl3_|-3.2.27_|-0.1.fc23_|-x86_64_|-installed
libnl3-cli_|-3.2.27_|-0.1.fc23_|-x86_64_|-installed
libnm-gtk_|-1.0.6_|-5.fc23_|-x86_64_|-installed
libnotify_|-0.7.6_|-7.fc23_|-x86_64_|-installed
liboauth_|-1.0.3_|-4.fc23_|-x86_64_|-installed
libodfgen_|-0.1.4_|-4.fc23_|-x86_64_|-installed
libogg_|-1.3.2_|-4.fc23_|-x86_64_|-installed
liborcus_|-0.7.1_|-6.fc23_|-x86_64_|-installed
libosinfo_|-0.2.12_|-2.fc23_|-x86_64_|-installed
libpagemaker_|-0.0.2_|-6.fc23_|-x86_64_|-installed
libpaper_|-1.1.24_|-11.fc23_|-x86_64_|-installed
libpath_utils_|-0.2.1_|-27.fc23_|-x86_64_|-installed
libpcap_|-1.7.4_|-1.fc23_|-x86_64_|-installed
libpciaccess_|-0.13.4_|-2.fc23_|-x86_64_|-installed
libpeas_|-1.16.0_|-1.fc23_|-x86_64_|-installed
libpeas-loader-python3_|-1.16.0_|-1.fc23_|-x86_64_|-installed
libphodav_|-2.0_|-5.fc23_|-x86_64_|-installed
libpinyin_|-1.2.0_|-1.fc23_|-x86_64_|-installed
libpinyin-data_|-1.2.0_|-1.fc23_|-x86_64_|-installed
libpipeline_|-1.4.0_|-2.fc23_|-x86_64_|-installed
libplist_|-1.12_|-5.fc23_|-x86_64_|-installed
libpng_|-1.6.17_|-2.fc23_|-x86_64_|-installed
libpng12_|-1.2.50_|-9.fc23_|-x86_64_|-installed
libproxy_|-0.4.11_|-12.fc23_|-x86_64_|-installed
libproxy-mozjs_|-0.4.11_|-12.fc23_|-x86_64_|-installed
libpsl_|-0.7.0_|-5.fc23_|-x86_64_|-installed
libpurple_|-2.10.11_|-14.fc23_|-x86_64_|-installed
libpwquality_|-1.3.0_|-2.fc23_|-x86_64_|-installed
libqb_|-0.17.2_|-1.fc23_|-x86_64_|-installed
libqmi_|-1.12.6_|-1.fc23_|-x86_64_|-installed
libqmi-utils_|-1.12.6_|-1.fc23_|-x86_64_|-installed
libquvi_|-0.9.4_|-7.fc23_|-x86_64_|-installed
libquvi-scripts_|-0.9.20131130_|-5.fc23_|-noarch_|-installed
librados2_|-0.94.5_|-1.fc23_|-x86_64_|-installed
libraw1394_|-2.1.0_|-6.fc23_|-x86_64_|-installed
librbd1_|-0.94.5_|-1.fc23_|-x86_64_|-installed
librdmacm_|-1.0.18.1_|-4.fc23_|-x86_64_|-installed
libref_array_|-0.1.5_|-27.fc23_|-x86_64_|-installed
libreoffice-calc_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-core_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-draw_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-emailmerge_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-filters_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-graphicfilter_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-impress_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-math_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-opensymbol-fonts_|-5.0.3.2_|-4.fc23_|-noarch_|-installed
libreoffice-pdfimport_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-pyuno_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-ure_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-writer_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
libreoffice-writer2latex_|-1.0.2_|-17.fc23_|-x86_64_|-installed
libreoffice-xsltfilter_|-5.0.3.2_|-4.fc23_|-x86_64_|-installed
librepo_|-1.7.16_|-2.fc23_|-x86_64_|-installed
libreport_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-anaconda_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-cli_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-fedora_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-filesystem_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-gtk_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-plugin-bugzilla_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-plugin-kerneloops_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-plugin-logger_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-plugin-reportuploader_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-plugin-ureport_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-python_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-python3_|-2.6.3_|-1.fc23_|-x86_64_|-installed
libreport-web_|-2.6.3_|-1.fc23_|-x86_64_|-installed
librevenge_|-0.0.2_|-6.fc23_|-x86_64_|-installed
librsvg2_|-2.40.11_|-1.fc23_|-x86_64_|-installed
libsamplerate_|-0.1.8_|-9.fc23_|-x86_64_|-installed
libsane-hpaio_|-3.15.9_|-2.fc23_|-x86_64_|-installed
libseccomp_|-2.2.3_|-0.fc23_|-x86_64_|-installed
libsecret_|-0.18.3_|-1.fc23_|-x86_64_|-installed
libselinux_|-2.4_|-4.fc23_|-x86_64_|-installed
libselinux-devel_|-2.4_|-4.fc23_|-x86_64_|-installed
libselinux-python_|-2.4_|-4.fc23_|-x86_64_|-installed
libselinux-python3_|-2.4_|-4.fc23_|-x86_64_|-installed
libselinux-utils_|-2.4_|-4.fc23_|-x86_64_|-installed
libsemanage_|-2.4_|-4.fc23_|-x86_64_|-installed
libsemanage-python_|-2.4_|-4.fc23_|-x86_64_|-installed
libsemanage-python3_|-2.4_|-4.fc23_|-x86_64_|-installed
libsepol_|-2.4_|-1.fc23_|-x86_64_|-installed
libsepol-devel_|-2.4_|-1.fc23_|-x86_64_|-installed
libshout_|-2.2.2_|-13.fc23_|-x86_64_|-installed
libsigc++20_|-2.6.1_|-1.fc23_|-x86_64_|-installed
libsilc_|-1.1.10_|-14.fc23_|-x86_64_|-installed
libsmartcols_|-2.27.1_|-1.fc23_|-x86_64_|-installed
libsmbclient_|-4.3.0_|-3.fc23_|-x86_64_|-installed
libsndfile_|-1.0.25_|-17.fc23_|-x86_64_|-installed
libsodium_|-1.0.5_|-1.fc23_|-x86_64_|-installed
libsolv_|-0.6.14_|-2.fc23_|-x86_64_|-installed
libsoup_|-2.52.1_|-1.fc23_|-x86_64_|-installed
libspectre_|-0.2.7_|-7.fc23_|-x86_64_|-installed
libsrtp_|-1.5.0_|-3.fc23_|-x86_64_|-installed
libss_|-1.42.13_|-3.fc23_|-x86_64_|-installed
libssh2_|-1.6.0_|-3.fc23_|-x86_64_|-installed
libsss_idmap_|-1.13.1_|-3.fc23_|-x86_64_|-installed
libsss_nss_idmap_|-1.13.1_|-3.fc23_|-x86_64_|-installed
libstdc++_|-5.1.1_|-4.fc23_|-x86_64_|-installed
libsysfs_|-2.1.0_|-18.fc23_|-x86_64_|-installed
libtalloc_|-2.1.3_|-1.fc23_|-x86_64_|-installed
libtar_|-1.2.20_|-7.fc23_|-x86_64_|-installed
libtasn1_|-4.5_|-2.fc23_|-x86_64_|-installed
libtdb_|-1.3.7_|-1.fc23_|-x86_64_|-installed
libteam_|-1.22_|-1.fc23_|-x86_64_|-installed
libtevent_|-0.9.25_|-2.fc23_|-x86_64_|-installed
libthai_|-0.1.21_|-2.fc23_|-x86_64_|-installed
libtheora_|-1.1.1_|-13.fc23_|-x86_64_|-installed
libtiff_|-4.0.4_|-1.fc23_|-x86_64_|-installed
libtimezonemap_|-0.4.4_|-2.fc23_|-x86_64_|-installed
libtirpc_|-0.3.2_|-4.rc3.fc23_|-x86_64_|-installed
libtomcrypt_|-1.17_|-25.fc23_|-x86_64_|-installed
libtommath_|-0.42.0_|-6.fc23_|-x86_64_|-installed
libtool-ltdl_|-2.4.6_|-5.fc23_|-x86_64_|-installed
libtranslit_|-0.0.3_|-9.fc23_|-x86_64_|-installed
libtranslit-m17n_|-0.0.3_|-9.fc23_|-x86_64_|-installed
libudisks2_|-2.1.6_|-1.fc23_|-x86_64_|-installed
libunistring_|-0.9.4_|-2.fc23_|-x86_64_|-installed
libunwind_|-1.1_|-10.fc23_|-x86_64_|-installed
libusal_|-1.1.11_|-29.fc23_|-x86_64_|-installed
libusb_|-0.1.5_|-6.fc23_|-x86_64_|-installed
libusbmuxd_|-1.0.10_|-3.fc23_|-x86_64_|-installed
libusbx_|-1.0.20_|-1.fc23_|-x86_64_|-installed
libuser_|-0.62_|-1.fc23_|-x86_64_|-installed
libuser-python_|-0.62_|-1.fc23_|-x86_64_|-installed
libutempter_|-1.1.6_|-7.fc23_|-x86_64_|-installed
libuuid_|-2.27.1_|-1.fc23_|-x86_64_|-installed
libv4l_|-1.8.1_|-1.fc23_|-x86_64_|-installed
libvdpau_|-1.1.1_|-2.fc23_|-x86_64_|-installed
libverto_|-0.2.6_|-5.fc23_|-x86_64_|-installed
libverto-devel_|-0.2.6_|-5.fc23_|-x86_64_|-installed
libverto-tevent_|-0.2.6_|-5.fc23_|-x86_64_|-installed
libvirt-client_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-config-network_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-interface_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-network_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-nodedev_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-nwfilter_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-qemu_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-secret_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-daemon-driver-storage_|-1.2.18.1_|-2.fc23_|-x86_64_|-installed
libvirt-gconfig_|-0.2.2_|-1.fc23_|-x86_64_|-installed
libvirt-glib_|-0.2.2_|-1.fc23_|-x86_64_|-installed
libvirt-gobject_|-0.2.2_|-1.fc23_|-x86_64_|-installed
libvisio_|-0.1.3_|-2.fc23_|-x86_64_|-installed
libvisual_|-0.4.0_|-18.fc23_|-x86_64_|-installed
libvorbis_|-1.3.4_|-5.fc23_|-x86_64_|-installed
libvpx_|-1.4.0_|-5.fc23_|-x86_64_|-installed
libwacom_|-0.15_|-3.fc23_|-x86_64_|-installed
libwacom-data_|-0.15_|-3.fc23_|-noarch_|-installed
libwayland-client_|-1.9.0_|-1.fc23_|-x86_64_|-installed
libwayland-cursor_|-1.9.0_|-1.fc23_|-x86_64_|-installed
libwayland-server_|-1.9.0_|-1.fc23_|-x86_64_|-installed
libwbclient_|-4.3.0_|-3.fc23_|-x86_64_|-installed
libwebp_|-0.4.4_|-1.fc23_|-x86_64_|-installed
libwinpr_|-1.2.0_|-0.9.git.24a752a.fc23_|-x86_64_|-installed
libwmf-lite_|-0.2.8.4_|-46.fc23_|-x86_64_|-installed
libwnck3_|-3.14.0_|-2.fc23_|-x86_64_|-installed
libwpd_|-0.10.0_|-5.fc23_|-x86_64_|-installed
libwpg_|-0.3.0_|-5.fc23_|-x86_64_|-installed
libwps_|-0.4.2_|-1.fc23_|-x86_64_|-installed
libwsman1_|-2.6.0_|-1.fc23_|-x86_64_|-installed
libwvstreams_|-4.6.1_|-14.fc23_|-x86_64_|-installed
libxcb_|-1.11.1_|-1.fc23_|-x86_64_|-installed
libxkbcommon_|-0.5.0_|-2.fc23_|-x86_64_|-installed
libxkbcommon-x11_|-0.5.0_|-2.fc23_|-x86_64_|-installed
libxkbfile_|-1.0.9_|-2.fc23_|-x86_64_|-installed
libxklavier_|-5.4_|-5.fc23_|-x86_64_|-installed
libxml++_|-2.40.0_|-1.fc23_|-x86_64_|-installed
libxml2_|-2.9.2_|-7.fc23_|-x86_64_|-installed
libxml2-devel_|-2.9.2_|-7.fc23_|-x86_64_|-installed
libxshmfence_|-1.2_|-2.fc23_|-x86_64_|-installed
libxslt_|-1.1.28_|-11.fc23_|-x86_64_|-installed
libyaml_|-0.1.6_|-7.fc23_|-x86_64_|-installed
libzapojit_|-0.0.3_|-7.fc23_|-x86_64_|-installed
linpsk_|-1.2_|-7.fc23_|-x86_64_|-installed
linux-atm-libs_|-2.5.1_|-13.fc23_|-x86_64_|-installed
linux-firmware_|-20151030_|-58.git66d3d8d7.fc23_|-noarch_|-installed
lklug-fonts_|-0.6_|-13.20090803cvs.fc23_|-noarch_|-installed
lksctp-tools_|-1.0.16_|-4.fc23_|-x86_64_|-installed
lldpad_|-1.0.1_|-2.git986eb2e.fc23_|-x86_64_|-installed
llvm-libs_|-3.7.0_|-1.fc23_|-x86_64_|-installed
lockdev_|-1.0.4_|-0.20.20111007git.fc23_|-x86_64_|-installed
log4net_|-1.2.13_|-5.fc23_|-x86_64_|-installed
logrotate_|-3.9.1_|-2.fc23_|-x86_64_|-installed
lohit-assamese-fonts_|-2.91.3_|-1.fc23_|-noarch_|-installed
lohit-bengali-fonts_|-2.91.3_|-1.fc23_|-noarch_|-installed
lohit-devanagari-fonts_|-2.95.2_|-1.fc23_|-noarch_|-installed
lohit-gujarati-fonts_|-2.92.2_|-4.fc23_|-noarch_|-installed
lohit-gurmukhi-fonts_|-2.91.0_|-6.fc23_|-noarch_|-installed
lohit-kannada-fonts_|-2.5.3_|-7.fc23_|-noarch_|-installed
lohit-odia-fonts_|-2.91.0_|-2.fc23_|-noarch_|-installed
lohit-tamil-fonts_|-2.91.1_|-2.fc23_|-noarch_|-installed
lohit-telugu-fonts_|-2.5.4_|-1.fc23_|-noarch_|-installed
lpsolve_|-5.5.2.0_|-11.fc23_|-x86_64_|-installed
lrzsz_|-0.12.20_|-38.fc23_|-x86_64_|-installed
lsof_|-4.88_|-3.fc23_|-x86_64_|-installed
lua_|-5.3.0_|-4.fc23_|-x86_64_|-installed
lua-expat_|-1.3.0_|-6.fc23_|-x86_64_|-installed
lua-json_|-1.3.2_|-5.fc23_|-noarch_|-installed
lua-lpeg_|-0.12.2_|-1.fc23_|-x86_64_|-installed
lua-socket_|-3.0_|-0.11.rc1.fc23_|-x86_64_|-installed
lvm2_|-2.02.130_|-1.fc23_|-x86_64_|-installed
lvm2-libs_|-2.02.130_|-1.fc23_|-x86_64_|-installed
lyx_|-2.1.4_|-4.fc23_|-x86_64_|-installed
lyx-common_|-2.1.4_|-4.fc23_|-noarch_|-installed
lyx-fonts_|-2.1.4_|-4.fc23_|-noarch_|-installed
lz4_|-r131_|-1.fc23_|-x86_64_|-installed
lzo_|-2.08_|-5.fc23_|-x86_64_|-installed
lzop_|-1.03_|-12.fc23_|-x86_64_|-installed
m17n-db_|-1.7.0_|-5.fc23_|-noarch_|-installed
m17n-lib_|-1.7.0_|-4.fc23_|-x86_64_|-installed
m2crypto_|-0.21.1_|-20.fc23_|-x86_64_|-installed
m4_|-1.4.17_|-8.fc23_|-x86_64_|-installed
mactel-boot_|-0.9_|-12.fc23_|-x86_64_|-installed
mailcap_|-2.1.45_|-1.fc23_|-noarch_|-installed
mailx_|-12.5_|-17.fc23_|-x86_64_|-installed
make_|-4.0_|-5.1.fc23_|-x86_64_|-installed
man-db_|-2.7.1_|-10.fc23_|-x86_64_|-installed
man-pages_|-4.00_|-5.fc23_|-noarch_|-installed
marisa_|-0.2.4_|-12.fc23_|-x86_64_|-installed
mcelog_|-101_|-2.9bfaad8f92c5.fc23_|-x86_64_|-installed
mcpp_|-2.7.2_|-15.fc23_|-x86_64_|-installed
mdadm_|-3.3.2_|-2.fc23_|-x86_64_|-installed
meanwhile_|-1.1.0_|-18.fc23_|-x86_64_|-installed
media-player-info_|-22_|-1.fc23_|-noarch_|-installed
memtest86+_|-5.01_|-8.fc22_|-x86_64_|-installed
mercurial_|-3.5.1_|-1.fc23_|-x86_64_|-installed
mesa-dri-drivers_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-filesystem_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libEGL_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libGL_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libGLES_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libGLU_|-9.0.0_|-9.fc23_|-x86_64_|-installed
mesa-libgbm_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libglapi_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libwayland-egl_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
mesa-libxatracker_|-11.0.4_|-1.20151105.fc23_|-x86_64_|-installed
microcode_ctl_|-2.1_|-9.1.fc23_|-x86_64_|-installed
midori_|-0.5.10_|-2.fc23_|-x86_64_|-installed
mlocate_|-0.26_|-12.fc23_|-x86_64_|-installed
mobile-broadband-provider-info_|-1.20150421git_|-2.fc23_|-noarch_|-installed
mod_dnssd_|-0.6_|-13.fc23_|-x86_64_|-installed
mokutil_|-0.2.0_|-3.fc23_|-x86_64_|-installed
mono-core_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-data_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-data-sqlite_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-extras_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-mvc_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-wcf_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-web_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-winforms_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mono-winfx_|-4.0.4_|-1.fc23_|-x86_64_|-installed
mousetweaks_|-3.12.0_|-4.fc23_|-x86_64_|-installed
mozilla-filesystem_|-1.9_|-13.fc23_|-x86_64_|-installed
mozjs17_|-17.0.0_|-12.fc23_|-x86_64_|-installed
mozjs24_|-24.2.0_|-5.fc22_|-x86_64_|-installed
mpage_|-2.5.6_|-16.fc23_|-x86_64_|-installed
mpfr_|-3.1.3_|-1.fc23_|-x86_64_|-installed
mtdev_|-1.1.5_|-5.fc23_|-x86_64_|-installed
mtools_|-4.0.18_|-9.fc23_|-x86_64_|-installed
mtr_|-0.85_|-7.fc23_|-x86_64_|-installed
mutter_|-3.18.1_|-4.fc23_|-x86_64_|-installed
mythes_|-1.2.4_|-4.fc23_|-x86_64_|-installed
nautilus_|-3.18.1_|-1.fc23_|-x86_64_|-installed
nautilus-dropbox_|-2015.10.28_|-1.fc10_|-x86_64_|-installed
nautilus-extensions_|-3.18.1_|-1.fc23_|-x86_64_|-installed
nautilus-sendto_|-3.8.2_|-2.fc23_|-x86_64_|-installed
naver-nanum-fonts-common_|-3.020_|-15.20140930.fc23_|-noarch_|-installed
naver-nanum-gothic-fonts_|-3.020_|-15.20140930.fc23_|-noarch_|-installed
ncurses_|-5.9_|-21.20150214.fc23_|-x86_64_|-installed
ncurses-base_|-5.9_|-21.20150214.fc23_|-noarch_|-installed
ncurses-libs_|-5.9_|-21.20150214.fc23_|-x86_64_|-installed
neon_|-0.30.1_|-3.fc23_|-x86_64_|-installed
net-snmp-libs_|-5.7.3_|-7.fc23_|-x86_64_|-installed
net-tools_|-2.0_|-0.35.20150915git.fc23_|-x86_64_|-installed
netcf-libs_|-0.2.8_|-3.fc23_|-x86_64_|-installed
nettle_|-3.1.1_|-4.fc23_|-x86_64_|-installed
newt_|-0.52.18_|-3.fc23_|-x86_64_|-installed
newt-python_|-0.52.18_|-3.fc23_|-x86_64_|-installed
newt-python3_|-0.52.18_|-3.fc23_|-x86_64_|-installed
nfs-utils_|-1.3.3_|-1.rc1.fc23_|-x86_64_|-installed
nini_|-1.1.0_|-8.fc23_|-x86_64_|-installed
nm-connection-editor_|-1.0.6_|-5.fc23_|-x86_64_|-installed
nmap_|-6.47_|-5.fc23_|-x86_64_|-installed
nmap-ncat_|-6.47_|-5.fc23_|-x86_64_|-installed
notify-sharp_|-0.4.0_|-0.25.20100411svn.fc23_|-x86_64_|-installed
npth_|-1.2_|-2.fc23_|-x86_64_|-installed
nspr_|-4.10.10_|-1.fc23_|-x86_64_|-installed
nss_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
nss-mdns_|-0.10_|-16.fc23_|-x86_64_|-installed
nss-softokn_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
nss-softokn-freebl_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
nss-sysinit_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
nss-tools_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
nss-util_|-3.20.1_|-1.0.fc23_|-x86_64_|-installed
ntfs-3g_|-2015.3.14_|-3.fc23_|-x86_64_|-installed
ntfsprogs_|-2015.3.14_|-3.fc23_|-x86_64_|-installed
numactl-libs_|-2.0.10_|-3.fc23_|-x86_64_|-installed
numad_|-0.5_|-20.20150602git.fc23_|-x86_64_|-installed
open-vm-tools_|-10.0.0_|-7.fc23_|-x86_64_|-installed
open-vm-tools-desktop_|-10.0.0_|-7.fc23_|-x86_64_|-installed
openCOLLADA_|-0_|-24.git3335ac1.fc23_|-x86_64_|-installed
openconnect_|-7.06_|-3.fc23_|-x86_64_|-installed
opengl-games-utils_|-0.2_|-7.fc23_|-noarch_|-installed
openjpeg2_|-2.1.0_|-7.fc23_|-x86_64_|-installed
openldap_|-2.4.40_|-14.fc23_|-x86_64_|-installed
openoffice.org-diafilter_|-1.7.4_|-1.fc23_|-x86_64_|-installed
openssh_|-7.1p1_|-5.fc23_|-x86_64_|-installed
openssh-askpass_|-7.1p1_|-5.fc23_|-x86_64_|-installed
openssh-clients_|-7.1p1_|-5.fc23_|-x86_64_|-installed
openssh-server_|-7.1p1_|-5.fc23_|-x86_64_|-installed
openssl_|-1.0.2d_|-2.fc23_|-x86_64_|-installed
openssl-devel_|-1.0.2d_|-2.fc23_|-x86_64_|-installed
openssl-libs_|-1.0.2d_|-2.fc23_|-x86_64_|-installed
openvpn_|-2.3.8_|-1.fc23_|-x86_64_|-installed
opus_|-1.1.1_|-0.4.rc.fc23_|-x86_64_|-installed
orc_|-0.4.24_|-1.fc23_|-x86_64_|-installed
orca_|-3.18.1_|-1.fc23_|-noarch_|-installed
os-prober_|-1.68_|-1.fc23_|-x86_64_|-installed
oxygen-icon-theme_|-5.16.0_|-1.fc23_|-noarch_|-installed
p11-kit_|-0.23.1_|-4.fc23_|-x86_64_|-installed
p11-kit-trust_|-0.23.1_|-4.fc23_|-x86_64_|-installed
pakchois_|-0.4_|-12.fc23_|-x86_64_|-installed
paktype-naskh-basic-fonts_|-4.1_|-5.fc23_|-noarch_|-installed
pam_|-1.2.1_|-2.fc23_|-x86_64_|-installed
pam_krb5_|-2.4.9_|-5.fc23_|-x86_64_|-installed
pam_pkcs11_|-0.6.8_|-6.fc23_|-x86_64_|-installed
pango_|-1.38.1_|-1.fc23_|-x86_64_|-installed
pangomm_|-2.38.1_|-1.fc23_|-x86_64_|-installed
paps_|-0.6.8_|-35.fc23_|-x86_64_|-installed
paps-libs_|-0.6.8_|-35.fc23_|-x86_64_|-installed
paratype-pt-sans-fonts_|-20141121_|-2.fc23_|-noarch_|-installed
parted_|-3.2_|-11.fc23_|-x86_64_|-installed
passwd_|-0.79_|-6.fc23_|-x86_64_|-installed
passwdqc_|-1.3.0_|-5.fc23_|-x86_64_|-installed
passwdqc-lib_|-1.3.0_|-5.fc23_|-x86_64_|-installed
patch_|-2.7.5_|-2.fc23_|-x86_64_|-installed
pciutils_|-3.3.1_|-2.fc23_|-x86_64_|-installed
pciutils-libs_|-3.3.1_|-2.fc23_|-x86_64_|-installed
pcmciautils_|-018_|-9.fc23_|-x86_64_|-installed
pcre_|-8.37_|-5.fc23_|-x86_64_|-installed
pcre-devel_|-8.37_|-5.fc23_|-x86_64_|-installed
pcsc-lite-libs_|-1.8.14_|-1.fc23_|-x86_64_|-installed
perl_|-5.22.0_|-349.fc23_|-x86_64_|-installed
perl-Carp_|-1.36_|-346.fc23_|-noarch_|-installed
perl-Data-Dumper_|-2.158_|-347.fc23_|-x86_64_|-installed
perl-Digest_|-1.17_|-346.fc23_|-noarch_|-installed
perl-Digest-MD5_|-2.54_|-346.fc23_|-x86_64_|-installed
perl-Encode_|-2.78_|-2.fc23_|-x86_64_|-installed
perl-Error_|-0.17024_|-4.fc23_|-noarch_|-installed
perl-Exporter_|-5.72_|-347.fc23_|-noarch_|-installed
perl-File-Path_|-2.09_|-347.fc23_|-noarch_|-installed
perl-File-Temp_|-0.23.04_|-346.fc23_|-noarch_|-installed
perl-Filter_|-1.55_|-1.fc23_|-x86_64_|-installed
perl-Getopt-Long_|-2.48_|-1.fc23_|-noarch_|-installed
perl-Git_|-2.5.0_|-2.fc23_|-noarch_|-installed
perl-HTTP-Tiny_|-0.056_|-3.fc23_|-noarch_|-installed
perl-MIME-Base64_|-3.15_|-348.fc23_|-x86_64_|-installed
perl-PathTools_|-3.56_|-346.fc23_|-x86_64_|-installed
perl-Pod-Escapes_|-1.07_|-348.fc23_|-noarch_|-installed
perl-Pod-Perldoc_|-3.25_|-347.fc23_|-noarch_|-installed
perl-Pod-Simple_|-3.31_|-1.fc23_|-noarch_|-installed
perl-Pod-Usage_|-1.67_|-3.fc23_|-noarch_|-installed
perl-Scalar-List-Utils_|-1.42_|-3.fc23_|-x86_64_|-installed
perl-Socket_|-2.020_|-1.fc23_|-x86_64_|-installed
perl-Storable_|-2.53_|-346.fc23_|-x86_64_|-installed
perl-Term-ANSIColor_|-4.03_|-346.fc23_|-noarch_|-installed
perl-Term-Cap_|-1.17_|-1.fc23_|-noarch_|-installed
perl-TermReadKey_|-2.33_|-3.fc23_|-x86_64_|-installed
perl-Text-ParseWords_|-3.30_|-346.fc23_|-noarch_|-installed
perl-Text-Tabs+Wrap_|-2013.0523_|-346.fc23_|-noarch_|-installed
perl-Text-Unidecode_|-1.27_|-1.fc23_|-noarch_|-installed
perl-Time-HiRes_|-1.9726_|-346.fc23_|-x86_64_|-installed
perl-Time-Local_|-1.2300_|-346.fc23_|-noarch_|-installed
perl-Tk_|-804.033_|-3.fc23_|-x86_64_|-installed
perl-Unicode-Normalize_|-1.23_|-1.fc23_|-x86_64_|-installed
perl-XML-Parser_|-2.44_|-3.fc23_|-x86_64_|-installed
perl-XML-XPath_|-1.13_|-28.fc23_|-noarch_|-installed
perl-constant_|-1.33_|-347.fc23_|-noarch_|-installed
perl-encoding_|-2.17_|-2.fc23_|-x86_64_|-installed
perl-libs_|-5.22.0_|-349.fc23_|-x86_64_|-installed
perl-macros_|-5.22.0_|-349.fc23_|-x86_64_|-installed
perl-open_|-1.10_|-349.fc23_|-noarch_|-installed
perl-parent_|-0.234_|-3.fc23_|-noarch_|-installed
perl-podlators_|-2.5.3_|-347.fc23_|-noarch_|-installed
perl-threads_|-2.02_|-2.fc23_|-x86_64_|-installed
perl-threads-shared_|-1.48_|-346.fc23_|-x86_64_|-installed
pigz_|-2.3.3_|-3.fc23_|-x86_64_|-installed
pinentry_|-0.9.6_|-4.fc23_|-x86_64_|-installed
pinentry-gtk_|-0.9.6_|-4.fc23_|-x86_64_|-installed
pinfo_|-0.6.10_|-12.fc23_|-x86_64_|-installed
pixman_|-0.33.4_|-1.fc23_|-x86_64_|-installed
pkcs11-helper_|-1.11_|-7.fc23_|-x86_64_|-installed
pkgconfig_|-0.28_|-9.fc23_|-x86_64_|-installed
plymouth_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-core-libs_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-graphics-libs_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-plugin-label_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-plugin-two-step_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-scripts_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-system-theme_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
plymouth-theme-charge_|-0.8.9_|-11.2013.08.14.fc23_|-x86_64_|-installed
policycoreutils_|-2.4_|-15.fc23_|-x86_64_|-installed
policycoreutils-python-utils_|-2.4_|-15.fc23_|-x86_64_|-installed
policycoreutils-python3_|-2.4_|-15.fc23_|-x86_64_|-installed
polkit_|-0.113_|-4.fc23_|-x86_64_|-installed
polkit-gnome_|-0.105_|-9.fc23_|-x86_64_|-installed
polkit-libs_|-0.113_|-4.fc23_|-x86_64_|-installed
polkit-pkla-compat_|-0.1_|-6.fc23_|-x86_64_|-installed
poppler_|-0.34.0_|-1.fc23_|-x86_64_|-installed
poppler-data_|-0.4.7_|-4.fc23_|-noarch_|-installed
poppler-glib_|-0.34.0_|-1.fc23_|-x86_64_|-installed
poppler-utils_|-0.34.0_|-1.fc23_|-x86_64_|-installed
popt_|-1.16_|-6.fc23_|-x86_64_|-installed
ppp_|-2.4.7_|-8.fc23_|-x86_64_|-installed
pptp_|-1.8.0_|-7.fc23_|-x86_64_|-installed
procps-ng_|-3.3.10_|-9.fc23_|-x86_64_|-installed
protobuf_|-2.6.1_|-2.fc23_|-x86_64_|-installed
protobuf-c_|-1.1.1_|-2.fc23_|-x86_64_|-installed
protobuf-compiler_|-2.6.1_|-2.fc23_|-x86_64_|-installed
psacct_|-6.6.2_|-3.fc23_|-x86_64_|-installed
psmisc_|-22.21_|-7.fc23_|-x86_64_|-installed
pulseaudio_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-gdm-hooks_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-libs_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-libs-glib2_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-module-bluetooth_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-module-x11_|-7.1_|-1.fc23_|-x86_64_|-installed
pulseaudio-utils_|-7.1_|-1.fc23_|-x86_64_|-installed
pyatspi_|-2.18.0_|-1.fc23_|-noarch_|-installed
pycairo_|-1.10.0_|-2.fc23_|-x86_64_|-installed
pygobject2_|-2.28.6_|-14.fc23_|-x86_64_|-installed
pygpgme_|-0.3_|-13.fc23_|-x86_64_|-installed
pygtk2_|-2.24.0_|-12.fc23_|-x86_64_|-installed
pygtk2-libglade_|-2.24.0_|-12.fc23_|-x86_64_|-installed
pykickstart_|-2.13_|-1.fc23_|-noarch_|-installed
pyliblzma_|-0.5.3_|-14.fc23_|-x86_64_|-installed
pyparted_|-3.10.7_|-1.fc23_|-x86_64_|-installed
python_|-2.7.10_|-8.fc23_|-x86_64_|-installed
python-IPy_|-0.81_|-13.fc23_|-noarch_|-installed
python-IPy-python3_|-0.81_|-13.fc23_|-noarch_|-installed
python-augeas_|-0.5.0_|-3.fc23_|-noarch_|-installed
python-babel_|-1.3_|-8.fc23_|-noarch_|-installed
python-backports_|-1.0_|-6.fc23_|-x86_64_|-installed
python-backports-ssl_match_hostname_|-3.4.0.2_|-4.fc22_|-noarch_|-installed
python-blivet_|-1.12.8_|-1.fc23_|-noarch_|-installed
python-caribou_|-0.4.19_|-1.fc23_|-noarch_|-installed
python-chardet_|-2.2.1_|-3.fc23_|-noarch_|-installed
python-coverage_|-4.0_|-1.fc23_|-x86_64_|-installed
python-crypto_|-2.6.1_|-7.fc23_|-x86_64_|-installed
python-decorator_|-3.4.2_|-2.fc23_|-noarch_|-installed
python-dmidecode_|-3.12.2_|-1.fc23_|-x86_64_|-installed
python-dnf-plugins-core_|-0.1.13_|-1.fc23_|-noarch_|-installed
python-dnf-plugins-extras-common_|-0.0.11_|-1.fc23_|-noarch_|-installed
python-dnf-plugins-extras-migrate_|-0.0.11_|-1.fc23_|-noarch_|-installed
python-firewall_|-0.3.14.2_|-4.fc23_|-noarch_|-installed
python-gobject_|-3.18.0_|-2.fc23_|-x86_64_|-installed
python-gobject-base_|-3.18.0_|-2.fc23_|-x86_64_|-installed
python-hawkey_|-0.6.2_|-1.fc23_|-x86_64_|-installed
python-iniparse_|-0.4_|-16.fc23_|-noarch_|-installed
python-jinja2_|-2.8_|-2.fc23_|-noarch_|-installed
python-kickstart_|-2.13_|-1.fc23_|-noarch_|-installed
python-kitchen_|-1.2.1_|-3.fc23_|-noarch_|-installed
python-libcomps_|-0.1.7_|-1.fc23_|-x86_64_|-installed
python-librepo_|-1.7.16_|-2.fc23_|-x86_64_|-installed
python-libs_|-2.7.10_|-8.fc23_|-x86_64_|-installed
python-libxml2_|-2.9.2_|-7.fc23_|-x86_64_|-installed
python-markupsafe_|-0.23_|-7.fc23_|-x86_64_|-installed
python-meh_|-0.43_|-1.fc23_|-noarch_|-installed
python-meh-gui_|-0.43_|-1.fc23_|-noarch_|-installed
python-msgpack_|-0.4.6_|-3.fc23_|-x86_64_|-installed
python-ntplib_|-0.3.3_|-1.fc23_|-noarch_|-installed
python-pillow_|-3.0.0_|-1.fc23_|-x86_64_|-installed
python-pip_|-7.1.0_|-1.fc23_|-noarch_|-installed
python-pwquality_|-1.3.0_|-2.fc23_|-x86_64_|-installed
python-pycurl_|-7.19.5.1_|-4.fc23_|-x86_64_|-installed
python-requests_|-2.7.0_|-7.fc23_|-noarch_|-installed
python-setuptools_|-18.0.1_|-2.fc23_|-noarch_|-installed
python-six_|-1.9.0_|-3.fc23_|-noarch_|-installed
python-slip_|-0.6.4_|-1.fc23_|-noarch_|-installed
python-slip-dbus_|-0.6.4_|-1.fc23_|-noarch_|-installed
python-sssdconfig_|-1.13.1_|-3.fc23_|-noarch_|-installed
python-systemd_|-230_|-1.fc23_|-x86_64_|-installed
python-urlgrabber_|-3.10.1_|-7.fc23_|-noarch_|-installed
python-urllib3_|-1.10.4_|-6.fc23_|-noarch_|-installed
python-xpyb_|-1.3.1_|-5.fc23_|-x86_64_|-installed
python-zmq_|-14.7.0_|-1.fc23_|-x86_64_|-installed
python2-dnf_|-1.1.3_|-1.fc23_|-noarch_|-installed
python2-dnf-plugin-system-upgrade_|-0.7.0_|-1.fc23_|-noarch_|-installed
python2-pyudev_|-0.17_|-2.fc23_|-noarch_|-installed
python3_|-3.4.3_|-5.fc23_|-x86_64_|-installed
python3-PyGithub_|-1.25.2_|-3.fc23_|-noarch_|-installed
python3-PyXB_|-1.2.4_|-5.fc23_|-noarch_|-installed
python3-PyYAML_|-3.11_|-9.fc23_|-x86_64_|-installed
python3-augeas_|-0.5.0_|-3.fc23_|-noarch_|-installed
python3-beaker_|-1.5.4_|-11.fc23_|-noarch_|-installed
python3-blivet_|-1.12.8_|-1.fc23_|-noarch_|-installed
python3-brlapi_|-0.6.3_|-10.fc23_|-x86_64_|-installed
python3-cairo_|-1.10.0_|-12.fc23_|-x86_64_|-installed
python3-caribou_|-0.4.19_|-1.fc23_|-noarch_|-installed
python3-chardet_|-2.2.1_|-3.fc23_|-noarch_|-installed
python3-coverage_|-4.0_|-1.fc23_|-x86_64_|-installed
python3-cssselect_|-0.9.1_|-6.fc23_|-noarch_|-installed
python3-cups_|-1.9.72_|-2.fc23_|-x86_64_|-installed
python3-dapp_|-0.3.0_|-2.fc23_|-noarch_|-installed
python3-dbus_|-1.2.0_|-8.fc23_|-x86_64_|-installed
python3-decorator_|-3.4.2_|-2.fc23_|-noarch_|-installed
python3-dmidecode_|-3.12.2_|-1.fc23_|-x86_64_|-installed
python3-dnf_|-1.1.3_|-1.fc23_|-noarch_|-installed
python3-dnf-plugin-system-upgrade_|-0.7.0_|-1.fc23_|-noarch_|-installed
python3-dnf-plugins-core_|-0.1.13_|-1.fc23_|-noarch_|-installed
python3-enchant_|-1.6.6_|-2.fc23_|-noarch_|-installed
python3-firewall_|-0.3.14.2_|-4.fc23_|-noarch_|-installed
python3-gobject_|-3.18.0_|-2.fc23_|-x86_64_|-installed
python3-gobject-base_|-3.18.0_|-2.fc23_|-x86_64_|-installed
python3-hawkey_|-0.6.2_|-1.fc23_|-x86_64_|-installed
python3-humanize_|-0.5.1_|-2.fc23_|-noarch_|-installed
python3-iniparse_|-0.4_|-16.fc23_|-noarch_|-installed
python3-inotify_|-0.9.6_|-2.fc23_|-noarch_|-installed
python3-iscsi-initiator-utils_|-6.2.0.873_|-29.git4c9d6f9.fc23_|-x86_64_|-installed
python3-javapackages_|-4.6.0_|-7.fc23_|-noarch_|-installed
python3-jinja2_|-2.8_|-2.fc23_|-noarch_|-installed
python3-kickstart_|-2.13_|-1.fc23_|-noarch_|-installed
python3-libcomps_|-0.1.7_|-1.fc23_|-x86_64_|-installed
python3-librepo_|-1.7.16_|-2.fc23_|-x86_64_|-installed
python3-libs_|-3.4.3_|-5.fc23_|-x86_64_|-installed
python3-libxml2_|-2.9.2_|-7.fc23_|-x86_64_|-installed
python3-louis_|-2.6.2_|-3.fc23_|-noarch_|-installed
python3-lxml_|-3.4.4_|-1.fc23_|-x86_64_|-installed
python3-mako_|-1.0.1_|-2.fc23_|-noarch_|-installed
python3-markupsafe_|-0.23_|-7.fc23_|-x86_64_|-installed
python3-meh_|-0.43_|-1.fc23_|-noarch_|-installed
python3-meh-gui_|-0.43_|-1.fc23_|-noarch_|-installed
python3-ntplib_|-0.3.3_|-1.fc23_|-noarch_|-installed
python3-pid_|-2.0.1_|-2.fc23_|-noarch_|-installed
python3-pillow_|-3.0.0_|-1.fc23_|-x86_64_|-installed
python3-pip_|-7.1.0_|-1.fc23_|-noarch_|-installed
python3-progress_|-1.2_|-5.fc23_|-noarch_|-installed
python3-pwquality_|-1.3.0_|-2.fc23_|-x86_64_|-installed
python3-pyatspi_|-2.18.0_|-1.fc23_|-noarch_|-installed
python3-pycurl_|-7.19.5.1_|-4.fc23_|-x86_64_|-installed
python3-pygpgme_|-0.3_|-13.fc23_|-x86_64_|-installed
python3-pyparted_|-3.10.7_|-1.fc23_|-x86_64_|-installed
python3-pytz_|-2015.4_|-1.fc23_|-noarch_|-installed
python3-pyudev_|-0.17_|-2.fc23_|-noarch_|-installed
python3-pyxdg_|-0.25_|-6.fc23_|-noarch_|-installed
python3-requests_|-2.7.0_|-7.fc23_|-noarch_|-installed
python3-requests-file_|-1.4_|-1.fc23_|-noarch_|-installed
python3-requests-ftp_|-0.3.1_|-1.fc23_|-noarch_|-installed
python3-setuptools_|-18.0.1_|-2.fc23_|-noarch_|-installed
python3-six_|-1.9.0_|-3.fc23_|-noarch_|-installed
python3-slip_|-0.6.4_|-1.fc23_|-noarch_|-installed
python3-slip-dbus_|-0.6.4_|-1.fc23_|-noarch_|-installed
python3-speechd_|-0.8.3_|-2.fc23_|-x86_64_|-installed
python3-sssdconfig_|-1.13.1_|-3.fc23_|-noarch_|-installed
python3-systemd_|-230_|-1.fc23_|-x86_64_|-installed
python3-urllib3_|-1.10.4_|-6.fc23_|-noarch_|-installed
pytz_|-2015.4_|-1.fc23_|-noarch_|-installed
pyxattr_|-0.5.3_|-5.fc23_|-x86_64_|-installed
qcustomplot-qt5_|-1.3.1_|-4.fc23_|-x86_64_|-installed
qemu-common_|-2.4.1_|-1.fc23_|-x86_64_|-installed
qemu-guest-agent_|-2.4.1_|-1.fc23_|-x86_64_|-installed
qemu-img_|-2.4.1_|-1.fc23_|-x86_64_|-installed
qhexedit2-qt5-libs_|-0.7.4_|-1.fc23_|-x86_64_|-installed
qpdf-libs_|-5.1.3_|-2.fc23_|-x86_64_|-installed
qrencode-libs_|-3.4.2_|-5.fc23_|-x86_64_|-installed
qscintilla-qt5_|-2.9_|-5.fc23_|-x86_64_|-installed
qt_|-4.8.7_|-3.fc23_|-x86_64_|-installed
qt-common_|-4.8.7_|-3.fc23_|-noarch_|-installed
qt-settings_|-23_|-8.fc23_|-noarch_|-installed
qt-x11_|-4.8.7_|-3.fc23_|-x86_64_|-installed
qt5-qtbase_|-5.5.0_|-18.fc23_|-x86_64_|-installed
qt5-qtbase-common_|-5.5.0_|-18.fc23_|-noarch_|-installed
qt5-qtbase-gui_|-5.5.0_|-18.fc23_|-x86_64_|-installed
qt5-qtdeclarative_|-5.5.0_|-2.fc23_|-x86_64_|-installed
qt5-qttools-common_|-5.5.0_|-4.fc23_|-noarch_|-installed
qt5-qttools-libs-designer_|-5.5.0_|-4.fc23_|-x86_64_|-installed
qt5-qtxmlpatterns_|-5.5.0_|-2.fc23_|-x86_64_|-installed
quota_|-4.02_|-4.fc23_|-x86_64_|-installed
quota-nls_|-4.02_|-4.fc23_|-noarch_|-installed
radvd_|-2.11_|-3.fc23_|-x86_64_|-installed
raptor2_|-2.0.15_|-4.fc23_|-x86_64_|-installed
rasqal_|-0.9.32_|-5.fc23_|-x86_64_|-installed
rdist_|-6.1.5_|-61.fc23_|-x86_64_|-installed
readline_|-6.3_|-6.fc23_|-x86_64_|-installed
realmd_|-0.16.2_|-1.fc23_|-x86_64_|-installed
recordmydesktop_|-0.3.8.1_|-15.fc23_|-x86_64_|-installed
redhat-lsb-core_|-4.1_|-30.fc23_|-x86_64_|-installed
redhat-lsb-submod-security_|-4.1_|-30.fc23_|-x86_64_|-installed
redhat-menus_|-12.0.2_|-9.fc23_|-noarch_|-installed
redland_|-1.0.17_|-4.fc23_|-x86_64_|-installed
rest_|-0.7.93_|-2.fc23_|-x86_64_|-installed
rhythmbox_|-3.2.1_|-3.fc23_|-x86_64_|-installed
rng-tools_|-5_|-5.fc23_|-x86_64_|-installed
rootfiles_|-8.1_|-18.fc23_|-noarch_|-installed
rp-pppoe_|-3.11_|-14.fc23_|-x86_64_|-installed
rpcbind_|-0.2.3_|-0.4.fc23_|-x86_64_|-installed
rpm_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-build-libs_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-libs_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-plugin-selinux_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-plugin-systemd-inhibit_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-python_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rpm-python3_|-4.13.0_|-0.rc1.6.fc23_|-x86_64_|-installed
rsync_|-3.1.1_|-7.fc23_|-x86_64_|-installed
rtkit_|-0.11_|-11.fc23_|-x86_64_|-installed
rubber_|-1.1_|-18.fc23_|-noarch_|-installed
rygel_|-0.28.1_|-1.fc23_|-x86_64_|-installed
salt_|-2015.5.5_|-1.fc23_|-noarch_|-installed
salt-minion_|-2015.5.5_|-1.fc23_|-noarch_|-installed
samba-client_|-4.3.0_|-3.fc23_|-x86_64_|-installed
samba-client-libs_|-4.3.0_|-3.fc23_|-x86_64_|-installed
samba-common_|-4.3.0_|-3.fc23_|-noarch_|-installed
samba-common-libs_|-4.3.0_|-3.fc23_|-x86_64_|-installed
sane-backends_|-1.0.25_|-1.fc23_|-x86_64_|-installed
sane-backends-drivers-scanners_|-1.0.25_|-1.fc23_|-x86_64_|-installed
sane-backends-libs_|-1.0.25_|-1.fc23_|-x86_64_|-installed
satyr_|-0.20_|-1.fc23_|-x86_64_|-installed
sbc_|-1.3_|-3.fc23_|-x86_64_|-installed
scl-utils_|-2.0.1_|-7.fc23_|-x86_64_|-installed
seabios-bin_|-1.8.2_|-1.fc23_|-noarch_|-installed
seahorse_|-3.18.0_|-2.fc23_|-x86_64_|-installed
seavgabios-bin_|-1.8.2_|-1.fc23_|-noarch_|-installed
sed_|-4.2.2_|-11.fc23_|-x86_64_|-installed
selinux-policy_|-3.13.1_|-154.fc23_|-noarch_|-installed
selinux-policy-targeted_|-3.13.1_|-154.fc23_|-noarch_|-installed
setools-libs_|-3.3.8_|-7.fc23_|-x86_64_|-installed
setroubleshoot_|-3.3.1_|-0.2.fc23_|-x86_64_|-installed
setroubleshoot-plugins_|-3.3.2_|-1.fc23_|-noarch_|-installed
setroubleshoot-server_|-3.3.1_|-0.2.fc23_|-x86_64_|-installed
setup_|-2.9.8_|-2.fc23_|-noarch_|-installed
setuptool_|-1.19.11_|-10.fc23_|-x86_64_|-installed
sg3_utils-libs_|-1.41_|-2.fc23_|-x86_64_|-installed
sgabios-bin_|-0.20110622svn_|-8.fc23_|-noarch_|-installed
sgpio_|-1.2.0.10_|-15.fc23_|-x86_64_|-installed
shadow-utils_|-4.2.1_|-3.fc23_|-x86_64_|-installed
shared-mime-info_|-1.5_|-2.fc23_|-x86_64_|-installed
sheepdog_|-0.3.0_|-10.fc23_|-x86_64_|-installed
shim_|-0.8_|-8_|-x86_64_|-installed
shotwell_|-0.22.0_|-5.fc23_|-x86_64_|-installed
sil-abyssinica-fonts_|-1.200_|-9.fc23_|-noarch_|-installed
sil-mingzat-fonts_|-0.100_|-4.fc23_|-noarch_|-installed
sil-nuosu-fonts_|-2.1.1_|-10.fc23_|-noarch_|-installed
sil-padauk-fonts_|-2.8_|-9.fc23_|-noarch_|-installed
skkdic_|-20150508_|-2.T1030.fc23_|-noarch_|-installed
slang_|-2.3.0_|-4.fc23_|-x86_64_|-installed
smc-fonts-common_|-6.1_|-5.fc23_|-noarch_|-installed
smc-meera-fonts_|-6.1_|-5.fc23_|-noarch_|-installed
smuxi_|-1.0_|-2.fc23_|-x86_64_|-installed
smuxi-engine_|-1.0_|-2.fc23_|-x86_64_|-installed
smuxi-frontend-gnome_|-1.0_|-2.fc23_|-x86_64_|-installed
snappy_|-1.1.3_|-1.fc23_|-x86_64_|-installed
sofia-sip_|-1.12.11_|-11.fc23_|-x86_64_|-installed
sofia-sip-glib_|-1.12.11_|-11.fc23_|-x86_64_|-installed
sos_|-3.2_|-0.3.a.fc23_|-noarch_|-installed
sound-theme-freedesktop_|-0.8_|-5.fc23_|-noarch_|-installed
soundtouch_|-1.4.0_|-13.fc23_|-x86_64_|-installed
soxr_|-0.1.2_|-1.fc23_|-x86_64_|-installed
spax_|-1.5.3_|-5.fc23_|-x86_64_|-installed
speech-dispatcher_|-0.8.3_|-2.fc23_|-x86_64_|-installed
speech-dispatcher-espeak_|-0.8.3_|-2.fc23_|-x86_64_|-installed
speex_|-1.2_|-0.24.rc2.fc23_|-x86_64_|-installed
speexdsp_|-1.2_|-0.7.rc3.fc23_|-x86_64_|-installed
spice-glib_|-0.30_|-1.fc23_|-x86_64_|-installed
spice-gtk3_|-0.30_|-1.fc23_|-x86_64_|-installed
spice-server_|-0.12.6_|-1.fc23_|-x86_64_|-installed
spice-vdagent_|-0.16.0_|-2.fc23_|-x86_64_|-installed
sqlite_|-3.9.0_|-1.fc23_|-x86_64_|-installed
sqlitebrowser_|-3.7.0_|-2.fc23_|-x86_64_|-installed
squirrel-libs_|-2.2.5_|-9.fc23_|-x86_64_|-installed
sssd_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-ad_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-client_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-common_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-common-pac_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-ipa_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-krb5_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-krb5-common_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-ldap_|-1.13.1_|-3.fc23_|-x86_64_|-installed
sssd-proxy_|-1.13.1_|-3.fc23_|-x86_64_|-installed
startup-notification_|-0.12_|-10.fc23_|-x86_64_|-installed
stix-fonts_|-1.1.0_|-8.fc23_|-noarch_|-installed
stoken-libs_|-0.90_|-2.fc23_|-x86_64_|-installed
stunnel_|-5.18_|-1.fc23_|-x86_64_|-installed
subunit_|-1.2.0_|-1.fc23_|-x86_64_|-installed
sudo_|-1.8.15_|-1.fc23_|-x86_64_|-installed
sushi_|-3.18.0_|-1.fc23_|-x86_64_|-installed
symlinks_|-1.4_|-12.fc23_|-x86_64_|-installed
syslinux_|-6.03_|-6.fc23_|-x86_64_|-installed
syslinux-extlinux_|-6.03_|-6.fc23_|-x86_64_|-installed
syslinux-extlinux-nonlinux_|-6.03_|-6.fc23_|-noarch_|-installed
syslinux-nonlinux_|-6.03_|-6.fc23_|-noarch_|-installed
system-config-printer-libs_|-1.5.7_|-5.fc23_|-noarch_|-installed
system-config-printer-udev_|-1.5.7_|-5.fc23_|-x86_64_|-installed
systemd_|-222_|-8.fc23_|-x86_64_|-installed
systemd-compat-libs_|-222_|-8.fc23_|-x86_64_|-installed
systemd-libs_|-222_|-8.fc23_|-x86_64_|-installed
tabish-eeyek-fonts_|-1.0_|-8.fc23_|-noarch_|-installed
taglib_|-1.9.1_|-10.fc23_|-x86_64_|-installed
tar_|-1.28_|-6.fc23_|-x86_64_|-installed
tcl_|-8.6.4_|-1.fc23_|-x86_64_|-installed
tcp_wrappers_|-7.6_|-81.fc23_|-x86_64_|-installed
tcp_wrappers-libs_|-7.6_|-81.fc23_|-x86_64_|-installed
tcpdump_|-4.7.4_|-3.fc23_|-x86_64_|-installed
tcplay_|-2.0_|-7.fc23_|-x86_64_|-installed
teamd_|-1.22_|-1.fc23_|-x86_64_|-installed
telepathy-farstream_|-0.6.1_|-6.fc23_|-x86_64_|-installed
telepathy-filesystem_|-0.0.2_|-8.fc23_|-noarch_|-installed
telepathy-gabble_|-0.18.2_|-5.fc23_|-x86_64_|-installed
telepathy-glib_|-0.24.1_|-2.fc23_|-x86_64_|-installed
telepathy-haze_|-0.8.0_|-3.fc22_|-x86_64_|-installed
telepathy-idle_|-0.2.0_|-4.fc23_|-x86_64_|-installed
telepathy-logger_|-0.8.2_|-2.fc23_|-x86_64_|-installed
telepathy-mission-control_|-5.16.3_|-3.fc23_|-x86_64_|-installed
telepathy-rakia_|-0.8.0_|-4.fc23_|-x86_64_|-installed
telepathy-salut_|-0.8.1_|-8.fc23_|-x86_64_|-installed
telnet_|-0.17_|-61.fc23_|-x86_64_|-installed
tetex-dvipost_|-1.1_|-20.fc23_|-x86_64_|-installed
texlive_|-2014_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-ae_|-svn15878.1.4_|-14.fc23_|-noarch_|-installed
texlive-algorithms_|-svn15878.0.1_|-14.fc23_|-noarch_|-installed
texlive-amscls_|-svn30738.0_|-14.fc23_|-noarch_|-installed
texlive-amsfonts_|-svn29208.3.04_|-14.fc23_|-noarch_|-installed
texlive-amsmath_|-svn30645.2.14_|-14.fc23_|-noarch_|-installed
texlive-anysize_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-attachfile_|-svn21866.v1.5b_|-14.fc23_|-noarch_|-installed
texlive-auto-pst-pdf_|-svn23723.0.6_|-14.fc23_|-noarch_|-installed
texlive-avantgar_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-babel_|-svn33283.3.9k_|-14.fc23_|-noarch_|-installed
texlive-babel-english_|-svn30264.0_|-14.fc23_|-noarch_|-installed
texlive-babelbib_|-svn25245.1.31_|-14.fc23_|-noarch_|-installed
texlive-base_|-2014_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-beamer_|-svn32491.3.33_|-14.fc23_|-noarch_|-installed
texlive-bera_|-svn20031.0_|-14.fc23_|-noarch_|-installed
texlive-beton_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-bibtex_|-svn29764.0.99d_|-14.fc23_|-noarch_|-installed
texlive-bibtex-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-bookman_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-booktabs_|-svn15878.1.61803_|-14.fc23_|-noarch_|-installed
texlive-breakurl_|-svn29901.1.40_|-14.fc23_|-noarch_|-installed
texlive-caption_|-svn30449.2013_05_12_|-14.fc23_|-noarch_|-installed
texlive-carlisle_|-svn18258.0_|-14.fc23_|-noarch_|-installed
texlive-charter_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-chktex_|-svn29821.1.7.1_|-14.fc23_|-noarch_|-installed
texlive-chktex-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-chngcntr_|-svn17157.1.0a_|-14.fc23_|-noarch_|-installed
texlive-cite_|-svn19955.5.3_|-14.fc23_|-noarch_|-installed
texlive-cm_|-svn32865.0_|-14.fc23_|-noarch_|-installed
texlive-cm-super_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-cmap_|-svn31477.1.0h_|-14.fc23_|-noarch_|-installed
texlive-cmextra_|-svn32831.0_|-14.fc23_|-noarch_|-installed
texlive-collection-basic_|-svn33040.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-collection-fontsrecommended_|-svn32620.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-collection-latex_|-svn33055.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-collection-latexrecommended_|-svn33055.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-colortbl_|-svn29803.v1.0a_|-14.fc23_|-noarch_|-installed
texlive-courier_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-crop_|-svn15878.1.5_|-14.fc23_|-noarch_|-installed
texlive-csquotes_|-svn24393.5.1d_|-14.fc23_|-noarch_|-installed
texlive-ctable_|-svn33011.1.28_|-14.fc23_|-noarch_|-installed
texlive-currfile_|-svn29012.0.7b_|-14.fc23_|-noarch_|-installed
texlive-dtl_|-svn29764.0.6.1_|-14.fc23_|-noarch_|-installed
texlive-dtl-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-dvipdfmx_|-svn32380.0_|-14.fc23_|-noarch_|-installed
texlive-dvipdfmx-bin_|-svn30845.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-dvipdfmx-def_|-svn33409.3.6_|-14.fc23_|-noarch_|-installed
texlive-dvips_|-svn33394.0_|-14.fc23_|-noarch_|-installed
texlive-dvips-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-ec_|-svn25033.1.0_|-14.fc23_|-noarch_|-installed
texlive-enctex_|-svn28602.0_|-14.fc23_|-noarch_|-installed
texlive-enumitem_|-svn24146.3.5.2_|-14.fc23_|-noarch_|-installed
texlive-environ_|-svn29600.0.3_|-14.fc23_|-noarch_|-installed
texlive-eso-pic_|-svn31875.2.0d_|-14.fc23_|-noarch_|-installed
texlive-etex_|-svn31600.2.1_|-14.fc23_|-noarch_|-installed
texlive-etex-pkg_|-svn15878.2.0_|-14.fc23_|-noarch_|-installed
texlive-etoolbox_|-svn20922.2.1_|-14.fc23_|-noarch_|-installed
texlive-euler_|-svn17261.2.5_|-14.fc23_|-noarch_|-installed
texlive-euro_|-svn22191.1.1_|-14.fc23_|-noarch_|-installed
texlive-eurosym_|-svn17265.1.4_subrfix_|-14.fc23_|-noarch_|-installed
texlive-extsizes_|-svn17263.1.4a_|-14.fc23_|-noarch_|-installed
texlive-fancybox_|-svn18304.1.4_|-14.fc23_|-noarch_|-installed
texlive-fancyhdr_|-svn15878.3.1_|-14.fc23_|-noarch_|-installed
texlive-fancyref_|-svn15878.0.9c_|-14.fc23_|-noarch_|-installed
texlive-fancyvrb_|-svn18492.2.8_|-14.fc23_|-noarch_|-installed
texlive-filecontents_|-svn24250.1.3_|-14.fc23_|-noarch_|-installed
texlive-filehook_|-svn24280.0.5d_|-14.fc23_|-noarch_|-installed
texlive-fix2col_|-svn17133.0_|-14.fc23_|-noarch_|-installed
texlive-float_|-svn15878.1.3d_|-14.fc23_|-noarch_|-installed
texlive-fontspec_|-svn32416.v2.3c_|-14.fc23_|-noarch_|-installed
texlive-footmisc_|-svn23330.5.5b_|-14.fc23_|-noarch_|-installed
texlive-fp_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-fpl_|-svn15878.1.002_|-14.fc23_|-noarch_|-installed
texlive-geometry_|-svn19716.5.6_|-14.fc23_|-noarch_|-installed
texlive-glyphlist_|-svn28576.0_|-14.fc23_|-noarch_|-installed
texlive-graphics_|-svn32655.1.0o_|-14.fc23_|-noarch_|-installed
texlive-gsftopk_|-svn29764.1.19.2_|-14.fc23_|-noarch_|-installed
texlive-gsftopk-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-helvetic_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-hyperref_|-svn28213.6.83m_|-14.fc23_|-noarch_|-installed
texlive-hyph-utf8_|-svn30757.0_|-14.fc23_|-noarch_|-installed
texlive-hyphen-base_|-svn31131.0_|-14.fc23_|-noarch_|-installed
texlive-ifetex_|-svn24853.1.2_|-14.fc23_|-noarch_|-installed
texlive-ifluatex_|-svn26725.1.3_|-14.fc23_|-noarch_|-installed
texlive-ifplatform_|-svn21156.0.3a_|-14.fc23_|-noarch_|-installed
texlive-iftex_|-svn29654.0.2_|-14.fc23_|-noarch_|-installed
texlive-ifxetex_|-svn19685.0.5_|-14.fc23_|-noarch_|-installed
texlive-index_|-svn24099.4.1beta_|-14.fc23_|-noarch_|-installed
texlive-jknapltx_|-svn19440.0_|-14.fc23_|-noarch_|-installed
texlive-kastrup_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-knuth-lib_|-svn33039.0_|-14.fc23_|-noarch_|-installed
texlive-knuth-local_|-svn33147.0_|-14.fc23_|-noarch_|-installed
texlive-koma-script_|-svn32489.3.12_|-14.fc23_|-noarch_|-installed
texlive-kpathsea_|-svn32266.0_|-14.fc23_|-noarch_|-installed
texlive-kpathsea-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-kpathsea-lib_|-2014_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-l3experimental_|-svn32537.SVN_4634_|-14.fc23_|-noarch_|-installed
texlive-l3kernel_|-svn32599.SVN_4646_|-14.fc23_|-noarch_|-installed
texlive-l3packages_|-svn32537.SVN_4634_|-14.fc23_|-noarch_|-installed
texlive-latex_|-svn32901.0_|-14.fc23_|-noarch_|-installed
texlive-latex-bin_|-svn29764.0_|-14.fc23_|-noarch_|-installed
texlive-latex-bin-bin_|-svn14050.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-latex-fonts_|-svn28888.0_|-14.fc23_|-noarch_|-installed
texlive-latexconfig_|-svn28991.0_|-14.fc23_|-noarch_|-installed
texlive-listings_|-svn33095.1.5c_|-14.fc23_|-noarch_|-installed
texlive-lm_|-svn28119.2.004_|-14.fc23_|-noarch_|-installed
texlive-lm-math_|-svn29044.1.958_|-14.fc23_|-noarch_|-installed
texlive-ltxmisc_|-svn21927.0_|-14.fc23_|-noarch_|-installed
texlive-lua-alt-getopt_|-svn29349.0.7.0_|-14.fc23_|-noarch_|-installed
texlive-lualatex-math_|-svn31389.1.3_|-14.fc23_|-noarch_|-installed
texlive-luaotfload_|-svn32887.2.4_3_|-14.fc23_|-noarch_|-installed
texlive-luaotfload-bin_|-svn30313.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-luatex_|-svn33312.0.70.1_|-14.fc23_|-noarch_|-installed
texlive-luatex-bin_|-svn31084.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-luatexbase_|-svn30562.0.6_|-14.fc23_|-noarch_|-installed
texlive-makeindex_|-svn33054.2.12_|-14.fc23_|-noarch_|-installed
texlive-makeindex-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-marginnote_|-svn25880.v1.1i_|-14.fc23_|-noarch_|-installed
texlive-marvosym_|-svn29349.2.2a_|-14.fc23_|-noarch_|-installed
texlive-mathpazo_|-svn15878.1.003_|-14.fc23_|-noarch_|-installed
texlive-mdwtools_|-svn15878.1.05.4_|-14.fc23_|-noarch_|-installed
texlive-memoir_|-svn31044.3.7b_|-14.fc23_|-noarch_|-installed
texlive-metafont_|-svn29764.2.718281_|-14.fc23_|-noarch_|-installed
texlive-metafont-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-metalogo_|-svn18611.0.12_|-14.fc23_|-noarch_|-installed
texlive-mflogo_|-svn32865.2.0_|-14.fc23_|-noarch_|-installed
texlive-mfnfss_|-svn19410.0_|-14.fc23_|-noarch_|-installed
texlive-mfware_|-svn32865.0_|-14.fc23_|-noarch_|-installed
texlive-mfware-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-mh_|-svn29420.0_|-14.fc23_|-noarch_|-installed
texlive-microtype_|-svn30755.2.5a_|-14.fc23_|-noarch_|-installed
texlive-mparhack_|-svn15878.1.4_|-14.fc23_|-noarch_|-installed
texlive-mptopdf_|-svn32580.0_|-14.fc23_|-noarch_|-installed
texlive-mptopdf-bin_|-svn18674.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-ms_|-svn29849.0_|-14.fc23_|-noarch_|-installed
texlive-multido_|-svn18302.1.42_|-14.fc23_|-noarch_|-installed
texlive-natbib_|-svn20668.8.31b_|-14.fc23_|-noarch_|-installed
texlive-ncntrsbk_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-ntgclass_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-oberdiek_|-svn33322.0_|-14.fc23_|-noarch_|-installed
texlive-palatino_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-paralist_|-svn30992.2.4_|-14.fc23_|-noarch_|-installed
texlive-parallel_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-parskip_|-svn19963.2.0_|-14.fc23_|-noarch_|-installed
texlive-pdfpages_|-svn31518.0.4v_|-14.fc23_|-noarch_|-installed
texlive-pdftex_|-svn33394.1.40.11_|-14.fc23_|-noarch_|-installed
texlive-pdftex-bin_|-svn30845.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-pdftex-def_|-svn22653.0.06d_|-14.fc23_|-noarch_|-installed
texlive-pgf_|-svn33057.3.0.0_|-14.fc23_|-noarch_|-installed
texlive-plain_|-svn26647.0_|-14.fc23_|-noarch_|-installed
texlive-powerdot_|-svn32549.1.5_|-14.fc23_|-noarch_|-installed
texlive-psfrag_|-svn15878.3.04_|-14.fc23_|-noarch_|-installed
texlive-pslatex_|-svn16416.0_|-14.fc23_|-noarch_|-installed
texlive-psnfss_|-svn23394.9.2a_|-14.fc23_|-noarch_|-installed
texlive-pspicture_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-pst-3d_|-svn17257.1.10_|-14.fc23_|-noarch_|-installed
texlive-pst-blur_|-svn15878.2.0_|-14.fc23_|-noarch_|-installed
texlive-pst-coil_|-svn24020.1.06_|-14.fc23_|-noarch_|-installed
texlive-pst-eps_|-svn15878.1.0_|-14.fc23_|-noarch_|-installed
texlive-pst-fill_|-svn15878.1.01_|-14.fc23_|-noarch_|-installed
texlive-pst-grad_|-svn15878.1.06_|-14.fc23_|-noarch_|-installed
texlive-pst-math_|-svn20176.0.61_|-14.fc23_|-noarch_|-installed
texlive-pst-node_|-svn33303.1.33_|-14.fc23_|-noarch_|-installed
texlive-pst-ovl_|-svn32514.0.04_|-14.fc23_|-noarch_|-installed
texlive-pst-pdf_|-svn31660.1.1v_|-14.fc23_|-noarch_|-installed
texlive-pst-pdf-bin_|-svn7838.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-pst-plot_|-svn33196.1.61_|-14.fc23_|-noarch_|-installed
texlive-pst-slpe_|-svn24391.1.31_|-14.fc23_|-noarch_|-installed
texlive-pst-text_|-svn15878.1.00_|-14.fc23_|-noarch_|-installed
texlive-pst-tree_|-svn24142.1.12_|-14.fc23_|-noarch_|-installed
texlive-pstricks_|-svn33328.2.53a_|-14.fc23_|-noarch_|-installed
texlive-pstricks-add_|-svn32760.3.65_|-14.fc23_|-noarch_|-installed
texlive-pxfonts_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-qstest_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-rcs_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-rotating_|-svn16832.2.16b_|-14.fc23_|-noarch_|-installed
texlive-rsfs_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-sansmath_|-svn17997.1.1_|-14.fc23_|-noarch_|-installed
texlive-sansmathaccent_|-svn30187.0_|-14.fc23_|-noarch_|-installed
texlive-sauerj_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-scheme-basic_|-svn25923.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-section_|-svn20180.0_|-14.fc23_|-noarch_|-installed
texlive-seminar_|-svn32762.1.61_|-14.fc23_|-noarch_|-installed
texlive-sepnum_|-svn20186.2.0_|-14.fc23_|-noarch_|-installed
texlive-setspace_|-svn24881.6.7a_|-14.fc23_|-noarch_|-installed
texlive-showexpl_|-svn32737.v0.3l_|-14.fc23_|-noarch_|-installed
texlive-simplecv_|-svn15878.1.6_|-14.fc23_|-noarch_|-installed
texlive-soul_|-svn15878.2.4_|-14.fc23_|-noarch_|-installed
texlive-subfig_|-svn15878.1.3_|-14.fc23_|-noarch_|-installed
texlive-symbol_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-tetex_|-svn33394.3.0_|-14.fc23_|-noarch_|-installed
texlive-tetex-bin_|-svn29741.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-tex_|-svn29764.3.1415926_|-14.fc23_|-noarch_|-installed
texlive-tex-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-tex-gyre_|-svn18651.2.004_|-14.fc23_|-noarch_|-installed
texlive-tex-gyre-math_|-svn29045.0_|-14.fc23_|-noarch_|-installed
texlive-texconfig_|-svn29764.0_|-14.fc23_|-noarch_|-installed
texlive-texconfig-bin_|-svn29741.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-texlive-common-doc_|-svn30524.0_|-14.fc23_|-noarch_|-installed
texlive-texlive-en-doc_|-svn32016.0_|-14.fc23_|-noarch_|-installed
texlive-texlive-msg-translations_|-svn33412.0_|-14.fc23_|-noarch_|-installed
texlive-texlive-scripts_|-svn33431.0_|-14.fc23_|-noarch_|-installed
texlive-texlive-scripts-bin_|-svn29741.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-texlive.infra_|-svn32920.0_|-14.fc23_|-noarch_|-installed
texlive-texlive.infra-bin_|-svn29741.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-textcase_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-thumbpdf_|-svn29725.3.15_|-14.fc23_|-noarch_|-installed
texlive-thumbpdf-bin_|-svn6898.0_|-14.20140525_r34255.fc23_|-noarch_|-installed
texlive-times_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-tipa_|-svn29349.1.3_|-14.fc23_|-noarch_|-installed
texlive-tools_|-svn33063.0_|-14.fc23_|-noarch_|-installed
texlive-trimspaces_|-svn15878.1.1_|-14.fc23_|-noarch_|-installed
texlive-txfonts_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-type1cm_|-svn21820.0_|-14.fc23_|-noarch_|-installed
texlive-typehtml_|-svn17134.0_|-14.fc23_|-noarch_|-installed
texlive-ucs_|-svn30472.2.2_|-14.fc23_|-noarch_|-installed
texlive-ulem_|-svn26785.0_|-14.fc23_|-noarch_|-installed
texlive-underscore_|-svn18261.0_|-14.fc23_|-noarch_|-installed
texlive-unicode-math_|-svn30504.0.7d_|-14.fc23_|-noarch_|-installed
texlive-url_|-svn32528.3.4_|-14.fc23_|-noarch_|-installed
texlive-utopia_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-varwidth_|-svn24104.0.92_|-14.fc23_|-noarch_|-installed
texlive-wasy_|-svn15878.0_|-14.fc23_|-noarch_|-installed
texlive-wasysym_|-svn15878.2.0_|-14.fc23_|-noarch_|-installed
texlive-xcolor_|-svn15878.2.11_|-14.fc23_|-noarch_|-installed
texlive-xdvi_|-svn30339.22.86_|-14.fc23_|-noarch_|-installed
texlive-xdvi-bin_|-svn30088.0_|-14.20140525_r34255.fc23_|-x86_64_|-installed
texlive-xkeyval_|-svn27995.2.6a_|-14.fc23_|-noarch_|-installed
texlive-xunicode_|-svn30466.0.981_|-14.fc23_|-noarch_|-installed
texlive-zapfchan_|-svn31835.0_|-14.fc23_|-noarch_|-installed
texlive-zapfding_|-svn31835.0_|-14.fc23_|-noarch_|-installed
thai-scalable-fonts-common_|-0.5.0_|-10.fc23_|-noarch_|-installed
thai-scalable-waree-fonts_|-0.5.0_|-10.fc23_|-noarch_|-installed
tigervnc-license_|-1.5.0_|-3.fc23_|-noarch_|-installed
tigervnc-server-minimal_|-1.5.0_|-3.fc23_|-x86_64_|-installed
time_|-1.7_|-48.fc23_|-x86_64_|-installed
timedatex_|-0.3_|-3.fc23_|-x86_64_|-installed
tinyxml_|-2.6.2_|-10.fc23_|-x86_64_|-installed
totem_|-3.18.1_|-1.fc23_|-x86_64_|-installed
totem-nautilus_|-3.18.1_|-1.fc23_|-x86_64_|-installed
totem-pl-parser_|-3.10.5_|-2.fc23_|-x86_64_|-installed
traceroute_|-2.0.20_|-4.fc23_|-x86_64_|-installed
tracker_|-1.6.0_|-1.fc23_|-x86_64_|-installed
transmission-common_|-2.84_|-8.fc23_|-x86_64_|-installed
transmission-gtk_|-2.84_|-8.fc23_|-x86_64_|-installed
tree_|-1.7.0_|-5.fc23_|-x86_64_|-installed
trousers-lib_|-0.3.13_|-5.fc23_|-x86_64_|-installed
tzdata_|-2015g_|-1.fc23_|-noarch_|-installed
tzdata-java_|-2015g_|-1.fc23_|-noarch_|-installed
uboot-tools_|-2015.07_|-5.fc23_|-x86_64_|-installed
udisks2_|-2.1.6_|-1.fc23_|-x86_64_|-installed
unbound-libs_|-1.5.5_|-1.fc23_|-x86_64_|-installed
unoconv_|-0.7_|-1.fc23_|-noarch_|-installed
unzip_|-6.0_|-23.fc23_|-x86_64_|-installed
upower_|-0.99.3_|-2.fc23_|-x86_64_|-installed
uqm_|-0.7.0_|-10.fc23_|-x86_64_|-installed
urw-fonts_|-2.4_|-21.fc23_|-noarch_|-installed
usb_modeswitch_|-2.2.5_|-1.fc23_|-x86_64_|-installed
usb_modeswitch-data_|-20151101_|-1.fc23_|-noarch_|-installed
usbmuxd_|-1.1.0_|-5.fc23_|-x86_64_|-installed
usbredir_|-0.7.1_|-1.fc23_|-x86_64_|-installed
usbutils_|-008_|-4.fc23_|-x86_64_|-installed
usermode_|-1.111_|-7.fc23_|-x86_64_|-installed
ustr_|-1.0.4_|-18.fc22_|-x86_64_|-installed
util-linux_|-2.27.1_|-1.fc23_|-x86_64_|-installed
vconfig_|-1.9_|-17.fc23_|-x86_64_|-installed
vim-common_|-7.4.827_|-1.fc23_|-x86_64_|-installed
vim-enhanced_|-7.4.827_|-1.fc23_|-x86_64_|-installed
vim-filesystem_|-7.4.827_|-1.fc23_|-x86_64_|-installed
vim-minimal_|-7.4.827_|-1.fc23_|-x86_64_|-installed
vinagre_|-3.18.1_|-1.fc23_|-x86_64_|-installed
vino_|-3.18.0_|-1.fc23_|-x86_64_|-installed
vlgothic-fonts_|-20141206_|-2.fc23_|-noarch_|-installed
volume_key-libs_|-0.3.9_|-8.fc23_|-x86_64_|-installed
vpnc_|-0.5.3_|-26.svn550.fc23_|-x86_64_|-installed
vpnc-script_|-20140805_|-3.gitdf5808b.fc23_|-noarch_|-installed
vte_|-0.28.2_|-16.fc23_|-x86_64_|-installed
vte-profile_|-0.42.1_|-1.fc23_|-x86_64_|-installed
vte291_|-0.42.1_|-1.fc23_|-x86_64_|-installed
vte3_|-0.36.5_|-1.fc23_|-x86_64_|-installed
wavpack_|-4.70.0_|-5.fc23_|-x86_64_|-installed
web-assets-filesystem_|-5_|-3.fc23_|-noarch_|-installed
webkitgtk_|-2.4.9_|-3.fc23_|-x86_64_|-installed
webkitgtk3_|-2.4.9_|-3.fc23_|-x86_64_|-installed
webkitgtk4_|-2.10.3_|-1.fc23_|-x86_64_|-installed
webrtc-audio-processing_|-0.1_|-10.fc23_|-x86_64_|-installed
wget_|-1.16.3_|-2.fc23_|-x86_64_|-installed
which_|-2.20_|-12.fc23_|-x86_64_|-installed
wireless-tools_|-29_|-13.1.fc23_|-x86_64_|-installed
words_|-3.0_|-24.fc23_|-noarch_|-installed
wpa_supplicant_|-2.4_|-6.fc23_|-x86_64_|-installed
wv_|-1.2.9_|-12.fc23_|-x86_64_|-installed
wvdial_|-1.61_|-13.fc23_|-x86_64_|-installed
wxBase_|-2.8.12_|-19.fc23_|-x86_64_|-installed
wxGTK_|-2.8.12_|-19.fc23_|-x86_64_|-installed
xalan-c_|-1.11.0_|-7.fc23_|-x86_64_|-installed
xcb-util_|-0.4.0_|-3.fc23_|-x86_64_|-installed
xcb-util-image_|-0.4.0_|-3.fc23_|-x86_64_|-installed
xcb-util-keysyms_|-0.4.0_|-2.fc23_|-x86_64_|-installed
xcb-util-renderutil_|-0.3.9_|-4.fc23_|-x86_64_|-installed
xcb-util-wm_|-0.4.1_|-6.fc23_|-x86_64_|-installed
xclip_|-0.12_|-10.20140209svn.fc23_|-x86_64_|-installed
xdg-user-dirs_|-0.15_|-7.fc23_|-x86_64_|-installed
xdg-user-dirs-gtk_|-0.10_|-8.fc23_|-x86_64_|-installed
xdg-utils_|-1.1.1_|-1.fc23_|-noarch_|-installed
xen-libs_|-4.5.1_|-14.fc23_|-x86_64_|-installed
xen-licenses_|-4.5.1_|-14.fc23_|-x86_64_|-installed
xerces-c_|-3.1.2_|-3.fc23_|-x86_64_|-installed
xfce4-taskmanager_|-1.1.0_|-4.fc23_|-x86_64_|-installed
xfsprogs_|-3.2.4_|-1.fc23_|-x86_64_|-installed
xkeyboard-config_|-2.15_|-2.fc23_|-noarch_|-installed
xml-common_|-0.6.3_|-43.fc23_|-noarch_|-installed
xml-security-c_|-1.6.1_|-10.fc23_|-x86_64_|-installed
xmlrpc-c_|-1.32.5_|-1908.svn2451.fc23_|-x86_64_|-installed
xmlrpc-c-client_|-1.32.5_|-1908.svn2451.fc23_|-x86_64_|-installed
xonotic_|-0.8.1_|-4.fc23_|-x86_64_|-installed
xonotic-data_|-0.8.1_|-1.fc23_|-noarch_|-installed
xorg-x11-drv-ati_|-7.6.0_|-0.4.20150729git5510cd6.fc23_|-x86_64_|-installed
xorg-x11-drv-evdev_|-2.9.99_|-2.20150807git66c997886.fc23_|-x86_64_|-installed
xorg-x11-drv-fbdev_|-0.4.3_|-23.fc23_|-x86_64_|-installed
xorg-x11-drv-intel_|-2.99.917_|-16.20150729.fc23_|-x86_64_|-installed
xorg-x11-drv-libinput_|-0.14.0_|-2.fc23_|-x86_64_|-installed
xorg-x11-drv-nouveau_|-1.0.12_|-0.3.fc23_|-x86_64_|-installed
xorg-x11-drv-openchrome_|-0.3.3_|-17.fc23_|-x86_64_|-installed
xorg-x11-drv-qxl_|-0.1.4_|-6.fc23_|-x86_64_|-installed
xorg-x11-drv-synaptics_|-1.8.3_|-1.fc23_|-x86_64_|-installed
xorg-x11-drv-vesa_|-2.3.2_|-23.fc23_|-x86_64_|-installed
xorg-x11-drv-vmmouse_|-13.1.0_|-2.fc23_|-x86_64_|-installed
xorg-x11-drv-vmware_|-13.0.2_|-10.20150211git8f0cf7c.fc23_|-x86_64_|-installed
xorg-x11-drv-wacom_|-0.30.0_|-3.fc23_|-x86_64_|-installed
xorg-x11-font-utils_|-7.5_|-29.fc23_|-x86_64_|-installed
xorg-x11-server-Xorg_|-1.18.0_|-1.fc23_|-x86_64_|-installed
xorg-x11-server-Xwayland_|-1.18.0_|-1.fc23_|-x86_64_|-installed
xorg-x11-server-common_|-1.18.0_|-1.fc23_|-x86_64_|-installed
xorg-x11-server-utils_|-7.7_|-17.fc23_|-x86_64_|-installed
xorg-x11-utils_|-7.5_|-20.fc23_|-x86_64_|-installed
xorg-x11-xauth_|-1.0.9_|-4.fc23_|-x86_64_|-installed
xorg-x11-xinit_|-1.3.4_|-10.fc23_|-x86_64_|-installed
xorg-x11-xkb-utils_|-7.7_|-16.fc23_|-x86_64_|-installed
xterm_|-318_|-2.fc23_|-x86_64_|-installed
xz_|-5.2.1_|-3.fc23_|-x86_64_|-installed
xz-devel_|-5.2.1_|-3.fc23_|-x86_64_|-installed
xz-libs_|-5.2.1_|-3.fc23_|-x86_64_|-installed
yajl_|-2.1.0_|-4.fc23_|-x86_64_|-installed
yelp_|-3.17.2_|-3.fc23_|-x86_64_|-installed
yelp-libs_|-3.17.2_|-3.fc23_|-x86_64_|-installed
yelp-xsl_|-3.18.1_|-1.fc23_|-noarch_|-installed
yum_|-3.4.3_|-507.fc23_|-noarch_|-installed
yum-metadata-parser_|-1.1.4_|-15.fc23_|-x86_64_|-installed
yum-utils_|-1.1.31_|-508.fc23_|-noarch_|-installed
zd1211-firmware_|-1.4_|-11.fc23_|-noarch_|-installed
zeitgeist-libs_|-0.9.16_|-1.fc23_|-x86_64_|-installed
zenity_|-3.18.1.1_|-1.fc23_|-x86_64_|-installed
zeromq_|-4.1.2_|-1.fc23_|-x86_64_|-installed
zip_|-3.0_|-15.fc23_|-x86_64_|-installed
zlib_|-1.2.8_|-9.fc23_|-x86_64_|-installed
zlib-devel_|-1.2.8_|-9.fc23_|-x86_64_|-installed
zsh_|-5.1.1_|-2.fc23_|-x86_64_|-installed
zziplib_|-0.13.62_|-6.fc23_|-x86_64_|-installed
[TRACE ] stderr:
Yum-utils package has been deprecated, use dnf instead.
See 'man yum2dnf' for more information.
[INFO ] Package vim-enhanced is already installed.
[INFO ] Completed state [vim-enhanced] at time 17:39:26.526406
[DEBUG ] File /var/cache/salt/minion/accumulator/139831235922768 does not exist, no need to cleanup.
[DEBUG ] LazyLoaded highstate.output
[TRACE ] data = {'local': {'pkg_|-vim_|-vim-enhanced_|-installed': {'comment': u'Package vim-enhanced is already installed.', 'name': 'vim-enhanced', 'start_time': '17:39:24.006911', 'result': True, 'duration': u'2519.495 ms', '__run_num__': 0, 'changes': {}}}}
local:
----------
ID: vim
Function: pkg.installed
Name: vim-enhanced
Result: True
Comment: Package vim-enhanced is already installed.
Started: 17:39:24.006911
Duration: 2519.495 ms
Changes:
Summary
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
3.92user 0.55system 1:20.95elapsed 5%CPU (0avgtext+0avgdata 89312maxresident)k
0inputs+3296outputs (0major+83161minor)pagefaults 0swapsSorry for the delay here. This is a really interesting issue.
After some thought, I think the recommendation to adjust ulimits on the minion is the right one, with a few caveats:
max_open_files as the configuration option. Let's leave it commented out in the minion config and it will only be applied if specifically enabled by the user.Sound good?
djneades
commented
8 years ago That sounds good to me, @cachedout, thank you!
cachedout
commented
8 years ago @dneades Sounds good. Take a look at #30218 please.
djneades
commented
8 years ago Thank you – I’ve provided a number of comments and test results on the PR!
stale[bot]
commented
6 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
djneades
commented
6 years ago I’m not sure that this issue was ever fixed, was it? If not, since it is rather a major problem, closing it would seem premature …
stale[bot]
commented
6 years ago Thank you for updating this issue. It is no longer marked as stale.
OrangeDog
commented
5 years ago To clarify, is the current workaround to lower the open file limit to 2048? I note that the systemd unit for 2018.3.3 has it set to 8192, or was that already the fix?
stromnet
commented
5 years ago FYI; I have had slowness issues on multiple FreeBSD 11.2 machines with salt 2019.2.0. I found some good workarounds in the linked issue #48773:
# https://github.com/saltstack/salt/issues/48773#issuecomment-443599880
disable_grains:
- esxi
disable_modules:
- vsphere
- esxi
- esxcliAdding disabled_grains & disabled_modules gave great performance boost in running pretty simple states, with a total execution time from ~15-30s down to ~3s for 2 local hosts.
OrangeDog
commented
5 years ago @stromnet that's got nothing to do with this issue.
arizvisa
commented
5 years ago Is this issue still relevant, or can this be closed?
The code related to ssds seems to be non-existent as its now based on checking the output of geom (I'm looking in salt.grains.disks), and I don't appear to have the same performance issue.
I realize FreeBSD is a kind of non-popular target, but I'm still a fan. So if this issue still exists, I can try to track it down...
djneades
commented
5 years ago @arizvisa The primary problem is unrelated to SSDs, but is due to this Python 2 bug.
See my earlier comment for a full analysis of the cause, and this comment for a suggested workaround.
Here is the issue still occurring on a FreeBSD 12.0 machine with a high ulimit:
# ulimit -n 3000000
# time salt-call --local test.ping
local:
True
real 0m9.568s
user 0m4.088s
sys 0m5.484s
# ulimit -n 2048
# time salt-call --local test.ping
local:
True
real 0m0.868s
user 0m0.615s
sys 0m0.262s
# salt --versions
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.12.3
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.16 (default, Jul 25 2019, 01:18:22)
python-gnupg: Not Installed
PyYAML: 5.1
PyZMQ: 18.1.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist:
locale: UTF-8
machine: amd64
release: 12.0-RELEASE-p9
system: FreeBSD
version: Not InstalledNote that the underlying Python bug is fixed in Python 3, so the problem does not occur with Salt when using that:
# ulimit -n 3000000
# time salt-call --local test.ping
local:
True
real 0m3.292s
user 0m3.081s
sys 0m0.227s
# ulimit -n 2048
# time salt-call --local test.ping
local:
True
real 0m3.229s
user 0m3.005s
sys 0m0.225s
# salt --versions
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.12.3
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.9 (default, Jul 25 2019, 01:18:58)
python-gnupg: Not Installed
PyYAML: 5.1
PyZMQ: 18.1.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist:
locale: UTF-8
machine: amd64
release: 12.0-RELEASE-p9
system: FreeBSD
version: Not InstalledFinally, note that the test.ping performance on Python 3 is significantly worse here than that on Python 2 with a lowish ulimit. This may itself be worthy of its own issue and further investigation.
arizvisa
commented
5 years ago Ah. Ok, my bad. I guess I totally skimmed through the comments between the version-reports. I realized why I'm not seeing it on my system is because I'm using a fork of Python2 with backports of some Python3 features.
But since this is a problem in a core library, and judging by how it's been abandoned, this isn't something that salt can really fix, right? This'll go away at the end of the year as I "believe" that's when salt will drop support for Python2 :(.
Until that happens though, there's https://github.com/google/python-subprocess32/. You might be able to drop it in place of Python2's subprocess as it has the Python3 patch that you mentioned integrated. Back when I scoped out merging Python3's subprocess, I remember resolving that the libraries are essentially backwards compatible.
Maybe the title can be renamed to reference that it only occurs on Python2. They really need some Python3 tag for bugs like these for tracking.
djneades
commented
5 years ago @arizvisa No problem; I appreciate the further comments. The workaround we deployed was to ensure that the FreeBSD user account used by salt-ssh has a low ulimit.
As you say, this should fall out of the wash when Salt has fully migrated to Python 3. We’re not able to switch to Salt on Python 3 ourselves yet, as we are still being bitten by some of the outstanding bugs.
driskell
commented
5 years ago Seeing this too - related to #44758
There was a fix attempted but never completed here: https://github.com/saltstack/salt/pull/46286
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
stale[bot]
commented
4 years ago Thank you for updating this issue. It is no longer marked as stale.
asomers-ax
commented
4 years ago Upstream has now merged the correct fix! So this issue should no longer be a problem, if you use a recent build of Python on FreeBSD.
https://svnweb.freebsd.org/ports?view=revision&revision=518640 applied the patch to FreeBSD's Python packages, and https://github.com/python/cpython/commit/162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 applied the patch to the main Python repository.
If you use FreeBSD's Python packages, the minimum versions containing the fix are 3.8.1_2, 3.7.5_1, 3.6.9_1, 3.5.9_3, or 2.7.17_1 .
asomers-ax
commented
4 years ago @waynew @sagetherage I think you should close this issue now, since it's been fixed upstream by Python.
Note: This issue was originally titled ‘Grains collection glacially slow on FreeBSD 10.1 machine with multiple SSDs’. It has been renamed to better match the problem determined by the course of investigation outlined below.
A simple
salt-sshtest.pingcall was taking over 16 seconds against a single physical FreeBSD host with 128 GB RAM and 4 SSDs. After some hunting, I was able to locate the problem as being in the collection of grains data.The issue is easily demonstrated locally on the host using
salt-call --local:That took nearly 12 seconds! The same call, but skipping grains:
Grains collection is thus taking over 11 seconds on this host.
Running
salt-callwith logging set toallshows a delay of several seconds before theDevice kern.disks: does not report itself as an SSDline is output, and there is also a perceptible delay before each of the subsequent disk-related lines is output:Observations:
salt-sshextremely unpleasant to use against this host.kern.disksis being queried as to whether it is an SSD, since it is not a disk device. This looks like a bug.ipcommand (which does not exist on FreeBSD) would seem sub-optimal.Here is the versions report for the host, which has the
py27-saltpkg installed:The four SSDs da0 - da3 are Intel 3500 series 800 GB drives connected via an LSI 2308 controller. da4 is a USB flash drive. Here’s the relevant
dmesgoutput:The operating system is a fresh stock FreeBSD 10.1 install with all updates applied: