search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.17k stars 5.48k forks source link

rpmdb error causes traceback #44331

Closed DaveQB closed 6 years ago

DaveQB commented 7 years ago

Description of Issue/Question

rpm database is in a bad state. Salt returns a traceback rather than an elegant message. I know it is an edge-case but I thought I would report you want to catch this error and present a better message to the user. Otherwise, feel free to close this Issue.

error: rpmdb: BDB0113 Thread/process 25361/140576677242688 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm

      ID: soe-packages
Function: pkg.installed
  Result: False
 Comment: An exception occurred in this state: Traceback (most recent call last):
            File "/usr/lib/python2.7/site-packages/salt/state.py", line 1843, in call
              **cdata['kwargs'])
            File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1795, in wrapper
              return f(*args, **kwargs)
            File "/usr/lib/python2.7/site-packages/salt/states/pkg.py", line 1631, in installed
              **kwargs)
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 1284, in install
              _available = list_repo_pkgs(*has_wildcards, byrepo=False, **kwargs)
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 792, in list_repo_pkgs
              repo_name for repo_name, repo_info in six.iteritems(list_repos())
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 2456, in list_repos
              basedirs = _normalize_basedir(basedir)
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 382, in _normalize_basedir
              basedir = _get_yum_config_value('reposdir')
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 359, in _get_yum_config_value
              conf = _get_yum_config()
            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 308, in _get_yum_config
              for name, value in six.iteritems(yb.conf):
            File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1078, in <lambda>
              conf = property(fget=lambda self: self._getConfig(),
            File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 349, in _getConfig
              startupconf = config.readStartupConfig(fn, root, releasever)
            File "/usr/lib/python2.7/site-packages/yum/config.py", line 1093, in readStartupConfig
              startupconf.distroverpkg)
            File "/usr/lib/python2.7/site-packages/yum/config.py", line 1235, in _getsysver
              raise Errors.YumBaseError("Error: " + str(e))
          YumBaseError: Error: rpmdb open failed
 Started: 17:29:53.115194
Duration: 28718.161 ms
 Changes:

Setup

soe-packages: pkg.installed:

Steps to Reproduce Issue

salt-call --local state.apply

Versions Report

Salt Version: Salt: 2017.7.2

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 1.5 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: Not Installed M2Crypto: 0.21.1 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.5 (default, May 3 2017, 07:55:04) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: redhat 7.4 Maipo locale: UTF-8 machine: x86_64 release: 3.10.0-693.2.2.el7.x86_64 system: Linux version: Red Hat Enterprise Linux Server 7.4 Maipo

gtmanfred commented 7 years ago

Thanks for reporting this, we absolutely would love to see this fixed. If you had time to submit a PR we would greatly appreciate it

https://docs.saltstack.com/en/latest/topics/development/contributing.html

Otherwise, I have tagged it as low hanging fruit for people getting started contributing to work on.

Thanks! Daniel

DaveQB commented 7 years ago

@gtmanfred Good idea. I know a little python. Do we have a mentor type program where I could ask a few questions to help get started with this? I am suspecting it is a try...except improvement.

gtmanfred commented 7 years ago

You are more than welcome to ask here. Though we have our conference this week, so I will be a bit slow to respond.

You are correct that you want a try except block, around this statement

            File "/usr/lib/python2.7/site-packages/salt/modules/yumpkg.py", line 308, in _get_yum_config
              for name, value in six.iteritems(yb.conf):

And you want to catch the YumBaseError which you will need to import from somewhere in the yum python module, which you should be able to find reference too in this file /usr/lib/python2.7/site-packages/yum/config.py

DaveQB commented 7 years ago

Thanks @gtmanfred

gtmanfred commented 7 years ago

You could also try our new slack community slack or our irc or salt-users mailling list are also helpful.

https://saltstackcommunity.herokuapp.com https://github.com/saltstack/salt#get-saltstack-support-and-help

On Mon, Oct 30, 2017 at 7:57 PM, DaveQB notifications@github.com wrote:

Thanks @gtmanfred https://github.com/gtmanfred

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/44331#issuecomment-340636872, or mute the thread https://github.com/notifications/unsubscribe-auth/AAssoQFaN7CMRBY9JhvctB-mN6sQCbmYks5sxn6YgaJpZM4QKvxE .

vvanarasi commented 7 years ago

@DaveQB , Could you please confirm if you started working on this change.

DaveQB commented 7 years ago

No I have not.

null

vvanarasi commented 7 years ago

Can i work on this?

DaveQB commented 7 years ago

Absolutely.

null

leeclemens commented 6 years ago

I was able to reproduce the issue with:

Salt Version:
           Salt: 2017.7.2

After some testing with the current version, I have been unable to reproduce the traceback being printed in Comment.

I was using a smaller state, but the yum corruption/error handling should be the same:

my_test_pkgs:
  pkg.installed:
    - pkgs:
      - bc
dd if=/dev/urandom bs=1M count=10 of=/var/lib/rpm/Packages

Similar corruption was performed to other files, but no tracebacks were produced with either version

          ID: my_test_pkgs
    Function: pkg.installed
      Result: False
     Comment: Error occurred installing package(s). Additional info follows:

              errors:
                  - Running scope as unit run-19066.scope.
                    error: rpmdb: BDB0196 Encrypted checksum: no encryption key specified
                    [snip]
                    error: rpmdb: BDB0196 Encrypted checksum: no encryption key specified
                    error: rpmdb: BDB0210 /var/lib/rpm/Packages: metadata page checksum error
                    error: cannot open Packages index using db5 - Invalid argument (22)
                    error: cannot open Packages database in /var/lib/rpm
                    CRITICAL:yum.main:

                    Error: rpmdb open failed
     Started: 19:35:43.600234
    Duration: 1355.263 ms
     Changes:   
Salt Version:
           Salt: 2018.3.0-2548-g3daf751

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: 3.2.1
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.6

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.21.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core