ZFS grain/module unable to be reloaded during state run

[ryanwalder]

Description of Issue/Question

Unable to reload the zfs module during a state run.

Expected behaviour

0. run state.apply
1. Install zfsutils-linux package
2. reload zfs module
3. create zpool
4. state.apply finishes

Actual behaviour

0. Installs zfsutils-linux package
1. Reloads module but does not enable (because zfs_available grain is not updated)
2. Fails the create zpool due to module not being loaded corectly
3. state.apply finishes
4. run state.apply again and it works

Debug log: https://www.hastebin.com/himapiladu

Looks like it's unable to reload and activate the module due to relying on the grain zfs_support which is not being updated, changing the order of reload_modules and reload_grains in the below formula has no effect.

Setup

zfs:
  pkg.installed:
    - pkgs: ['zfsutils-linux']
    - reload_grains: True
    - reload_modules: True

zfs-zpool-foo:
  zpool.present:
    - name: foo
    - layout: [{'mirror': ['/dev/sdb', '/dev/sdc']}, {'mirror': ['/dev/sdd', '/dev/sde']}, {'mirror': ['/dev/sdf', '/dev/sdg']}, {'log': ['mirror', '/dev/sdh', '/dev/sdi']}]
    - config: {'device_dir': 'None', 'force': True}
    - require:
      - pkg: zfs

Steps to Reproduce Issue

Use the above

Versions Report

Salt Version:
           Salt: 2018.3.0-n/a-7390b72 (installed via bootstrap `git 2018.3`)

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-87-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[ryanwalder]

@sjorge Probably needs tagging.

[sjorge]

Can you try again after you have removed @real_memoize from the is_supported function in salt.utils.zfs?

[ryanwalder]

So I edited /usr/lib/python2.7/dist-packages/salt/utils/zfs.py then removed /usr/lib/python2.7/dist-packages/salt/utils/zfs.pyc and started the salt-master. Assuming this works (in lieu of an actual salt dev environment).

Removing @real_memoize from above is_supported did not work (same formula as above).

[sjorge]

It needs to be done on the minion running the state. (Although that could be the same host in your case)

If that did not do it, i’m not really sure why it would not properly reload it then.

[sjorge]

@ch3ll (hope tagging works because i’m on my phone)

Any ideas? I’m not super familiar with how all the reloading of grains/modules works inside a state run.

[ryanwalder]

Ah, my bad. I did it on the master previously.

Just so my process is clear (if wrong please correct me, I have no real experience hacking on salt):

0. Fire up minion in vagrant and bootstrap salt using git 2018.3
1. stop minion process
2. edit /usr/lib/python2.7/dist-packages/salt/utils/zfs.py to remove @real_memoize from above is_supported (line 282)
3. rm /usr/lib/python2.7/dist-packages/salt/utils/zfs.pyc
4. run salt-call state.apply zfs (the above formula)

Doesn't work :(

Debug log (looks the same unless I'm missing something) : https://www.hastebin.com/korojenafu

[Ch3LL]

just to clarify is it working the second time you run the state.appy or only on the first run? I would suspect reloading the grains is all it would take, so I want to verify that its working on the next runs.

[ryanwalder]

That's correct @Ch3LL.

First run installs the package but does not reload the grains/module as expected with reload_grains/reload_modules

A second state.apply right after the first has finished (and installed the zfsutils-linux) package works as expected: The zfs_available grain now returns True which allows the module to load and the state.apply to run correctly.

[Ch3LL]

ping @saltstack/team-core i'm not seeing why this would not be reloading the grain. any ideas here?

[sjorge]

@Ch3LL maybe because the ZFS grains are in salt/grains/zfs.py and not salt/grains/core.py ?

[Ch3LL]

@ryanwalder can you try this fix:

diff --git a/salt/utils/zfs.py b/salt/utils/zfs.py
index 4988824..b37d552 100644
--- a/salt/utils/zfs.py
+++ b/salt/utils/zfs.py
@@ -279,7 +279,6 @@ def _command(source, command, flags=None, opts=None,
     return ' '.join(cmd)

-@real_memoize
 def is_supported():
     '''
     Check the system for ZFS support
@@ -302,7 +301,7 @@ def is_supported():
         on_supported_platform = True

     # Additional check for the zpool command
-    return (_zpool_cmd() and on_supported_platform) is True
+    return (salt.utils.which('zpool') and on_supported_platform) is True

@sjorge and i worked offline with this approach. I believe it will work for you :)

[ryanwalder]

@Ch3LL & @sjorge That works a treat. Great work!

[Ch3LL]

Thanks for testing it :) I pushed the PR in #47518