search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

Open ggiesen opened 1 year ago

ggiesen commented 1 year ago

Description Sometime around 3006.0 - 3006.2 I started getting errors on my mount.mounted states for NFS mounts:

----------
          ID: mount_/etc/salt/pki/master/minions
    Function: mount.mounted
        Name: /etc/salt/pki/master/minions
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2381, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1293, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/mount.py", line 242, in mounted
                  active = __salt__["mount.active"](extended=True)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/mount.py", line 277, in active
                  _active_mountinfo(ret)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/mount.py", line 59, in _active_mountinfo
                  __context__["disk.blkid"] = __salt__["disk.blkid"]()
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/context.py", line 78, in __getitem__
                  return self.value()[item]
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 349, in __getitem__
                  super().__getitem__(item)  # try to get the item from the dictionary
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/lazy.py", line 105, in __getitem__
                  raise KeyError(key)
              KeyError: 'disk.blkid'
     Started: 14:32:19.750162
    Duration: 34.566 ms
     Changes:

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

State:

  mount_/etc/salt/pki/master/minions:
    mount.mounted:
    - name: /etc/salt/pki/master/minions
    - device: 192.0.2.97:/etc/salt/pki/master/minions
    - fstype: nfs4
    - opts:
      - defaults
      - rw
      - nosuid
      - nodev
      - noexec
      - relatime
      - vers=4.1
      - rsize=65536
      - wsize=65536
      - namlen=255
      - hard
      - proto=tcp
      - timeo=600
      - retrans=2
      - sec=sys
      - local_lock=none
    - persist: true
    - mount: true

Steps to Reproduce the behavior Try to apply the above state

Expected behavior State would report no changes

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3006.3 Python Version: Python: 3.10.13 (main, Sep 6 2023, 02:11:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: 1.5.0 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 1.11.1 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.13.10 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 Salt Extensions: saltext.bitwarden: 0.0.1b15 saltext.vmware: 23.6.29.0rc1 System Versions: dist: almalinux 8.7 Stone Smilodon locale: utf-8 machine: x86_64 release: 4.18.0-425.19.2.el8_7.x86_64 system: Linux version: AlmaLinux 8.7 Stone Smilodon ```
ggiesen commented 1 year ago

My guess it it was broken by @nicholasmhughes PR here: https://github.com/saltstack/salt/pull/63104 which looks like it was merged for 3006.0

nicholasmhughes commented 1 year ago

stop pointing fingers @ggiesen :wink:

ggiesen commented 1 year ago

lol I just thought you might be the best guy to fix it ;)

ggiesen commented 1 year ago

FYI NFS mounts don't show up with the blkid command:

# blkid
/dev/sda1: UUID="8256a0b9-70ef-4798-bf83-58859b5702fc" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="3ba51781-01"
/dev/sda2: UUID="yVSzYj-cniL-R7KW-OwSq-bsQE-lHTz-s1NayN" TYPE="LVM2_member" PARTUUID="3ba51781-02"
/dev/mapper/almalinux-root: UUID="9d9ae71e-473b-4c61-b67c-29bd242c7fb1" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/almalinux-swap: UUID="a23fc7b6-590a-4925-a707-491c6edd78dc" TYPE="swap"
/dev/mapper/almalinux-tmp: UUID="2ee59e1c-2551-4807-8e97-2bca72d39d0f" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/almalinux-home: UUID="5b72135d-401d-4f05-8cab-882b24dc8604" BLOCK_SIZE="512" TYPE="xfs
nicholasmhughes commented 1 year ago

I don't see anywhere in that PR of mine where I changed disk.blkid itself. The error looks like the LazyLoader can't find that function, which usually happens when there's a syntax error in the execution module somewhere. Can you run disk.blkid by itself? Are you by chance overriding any execution modules from the Salt filesystem?

ggiesen commented 1 year ago

Very astute, you are correct in that disk.blkid is not working:

[DEBUG   ] Could not LazyLoad disk.blkid: 'disk.blkid' is not available.
'disk.blkid' is not available.

I'm not, as far as I'm aware, I'm not overriding any execution modules from Salt (unless perhaps the saltext.vmware extension is, but that wouldn't touch the disk module). I'm currently dealing with more relenv packaging disasters through so who the heck knows.

nicholasmhughes commented 1 year ago

Do any of the other disk functions work? What about disk.usage?

ggiesen commented 1 year ago

Nope.

nicholasmhughes commented 1 year ago

There's definitely something borked on the system(s). The disk module is either missing or corrupt, or alternatively, Python path issues are preventing some modules from being found. You mentioned relenv issues... are you packaging Salt yourself?

ggiesen commented 1 year ago

I think I found it:

# salt-call disk.blkid -l trace
...
[TRACE   ] Loaded disk as virtual vmware_disk
[DEBUG   ] The functions from module 'vmware_disk' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'vmc_vm_disks' are being loaded by dir() on the loaded module
...
[DEBUG   ] Could not LazyLoad disk.blkid: 'disk.blkid' is not available.
'disk.blkid' is not available.

# salt-pip uninstall saltext.vmware
Found existing installation: saltext.vmware 23.6.29.0rc1
Uninstalling saltext.vmware-23.6.29.0rc1:
  Would remove:
    /opt/saltstack/salt/extras-3.10/saltext.vmware-23.6.29.0rc1.dist-info/*
    /opt/saltstack/salt/extras-3.10/saltext/vmware/*
Proceed (Y/n)? y
  Successfully uninstalled saltext.vmware-23.6.29.0rc1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# salt-call disk.blkid
local:
    ----------
    /dev/mapper/almalinux-home:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            5b72135d-401d-4f05-8cab-882b24dc8604
    /dev/mapper/almalinux-root:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            9d9ae71e-473b-4c61-b67c-29bd242c7fb1
    /dev/mapper/almalinux-swap:
        ----------
        TYPE:
            swap
        UUID:
            a23fc7b6-590a-4925-a707-491c6edd78dc
    /dev/mapper/almalinux-tmp:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            2ee59e1c-2551-4807-8e97-2bca72d39d0f
    /dev/sda1:
        ----------
        BLOCK_SIZE:
            512
        PARTUUID:
            3ba51781-01
        TYPE:
            xfs
        UUID:
            8256a0b9-70ef-4798-bf83-58859b5702fc
    /dev/sda2:
        ----------
        PARTUUID:
            3ba51781-02
        TYPE:
            LVM2_member
        UUID:
            yVSzYj-cniL-R7KW-OwSq-bsQE-lHTz-s1NayN
nicholasmhughes commented 1 year ago

Well... that's terrible.

ggiesen commented 1 year ago

Yep, I really was hoping it was your fault because there'd be a much higher chance of being fixed in a non-galactic timeframe.