Windows Salt-Minion 2019.2.0 AMD64 un-handeled exception after installation

alexandrebgallant commented 5 years ago

Description of Issue

After a clean install of a Windows 10 Enterprise ver 1903, I installed Salt-Minion 2019.2.0 AMD64. The service keeps on crashing and restarting. I tried the version with Python2 of Python3, Same problem. The minion doesn't registers to the Master.

Setup

Brand new computer with Windows 10 Enterprise ver 1903 Salt-Minion 2019.2.0

Steps to Reproduce Issue

Unable to start correctly the service. Unable to launch the salt-minion with Debug. Please see below the StackTrace of the salt-minion-debug.bat

salt-minion-debug.bat output

PS C:\salt> .\salt-minion-debug.bat
Le service salt-minion n’est pas lancé.
Vous obtiendrez une aide supplémentaire en entrant NET HELPMSG 3521.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] Multiprocessing queue logging configured for the process running under PID: 752 at log level 10
[DEBUG   ] Marking 'is_ip' as a jinja filter
[DEBUG   ] Marking 'is_ipv4' as a jinja filter
[DEBUG   ] Marking 'is_ipv6' as a jinja filter
[DEBUG   ] Marking 'ipv4' as a jinja filter
[DEBUG   ] Marking 'ipv6' as a jinja filter
[DEBUG   ] Marking 'ipaddr' as a jinja filter
[DEBUG   ] Marking 'ip_host' as a jinja filter
[DEBUG   ] Marking 'network_hosts' as a jinja filter
[DEBUG   ] Marking 'network_size' as a jinja filter
[DEBUG   ] Marking 'gen_mac' as a jinja filter
[DEBUG   ] Marking 'mac_str_to_bytes' as a jinja filter
[DEBUG   ] Marking 'dns_check' as a jinja filter
[DEBUG   ] Multiprocessing queue logging configured for the process running under PID: 752 at log level 10
[DEBUG   ] Configuration file path: c:\salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Minion "confMTL510"
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Created pidfile: c:\salt\var\run\salt-minion.pid
[INFO    ] Starting up the Salt Minion
[DEBUG   ] AsyncEventPublisher PUB socket URI: 4510
[DEBUG   ] AsyncEventPublisher PULL socket URI: 4511
[INFO    ] Starting pull socket on 4511
[DEBUG   ] SaltEvent PUB socket URI: 4510
[DEBUG   ] SaltEvent PULL socket URI: 4511
[DEBUG   ] Initializing new IPCClient for path: 4510
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-minion", line 26, in <module>
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-minion", line 26, in <module>
    salt_minion()
  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 191, in salt_minion
    minion.start()
  File "C:\salt\bin\lib\site-packages\salt\cli\daemons.py", line 348, in start
    self.minion.tune_in()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1056, in tune_in
    self._spawn_minions()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1000, in _spawn_minions
    jid_queue=self.jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 972, in _create_minion_object
    jid_queue=jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1126, in __init__
    self.opts['grains'] = salt.loader.grains(opts)
  File "C:\salt\bin\lib\site-packages\salt\loader.py", line 772, in grains
    ret = funcs[key]()
  File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1613, in os_data
    if 'Server' in grains['osrelease']:
TypeError: argument of type 'NoneType' is not iterable    salt_minion()
  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 191, in salt_minion
    minion.start()
  File "C:\salt\bin\lib\site-packages\salt\cli\daemons.py", line 348, in start
    self.minion.tune_in()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1056, in tune_in
    self._spawn_minions()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1000, in _spawn_minions
    jid_queue=self.jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 972, in _create_minion_object
    jid_queue=jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1126, in __init__
    self.opts['grains'] = salt.loader.grains(opts)
  File "C:\salt\bin\lib\site-packages\salt\loader.py", line 772, in grains
    ret = funcs[key]()
  File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1613, in os_data
    if 'Server' in grains['osrelease']:
TypeError: argument of type 'NoneType' is not iterable

alexandrebgallant commented 5 years ago

I was able to diagnose further with one of my collegues.

The problem is in the minion "C:\salt\bin\Lib\sites-packages\salt\grains\core.py"

We commented out the whole If Statement, and setup a static os_release variable.

# This creates the osrelease grain based on the Windows Operating
        # System Product Name. As long as Microsoft maintains a similar format
        # this should be future proof
        version = 'Unknown'
        release = ''
        os_release = '10'
        # if 'Server' in osinfo.Caption:
            # for item in osinfo.Caption.split(' '):
                # # If it's all digits, then it's version
                # if re.match(r'\d+', item):
                    # version = item
                # # If it starts with R and then numbers, it's the release
                # # ie: R2
                # if re.match(r'^R\d+$', item):
                    # release = item
            # os_release = '{0}Server{1}'.format(version, release)
        # else:
            # for item in osinfo.Caption.split(' '):
                # # If it's a number, decimal number, Thin or Vista, then it's the
                # # version
                # if re.match(r'^(\d+(\.\d+)?)|Thin|Vista$', item):
                    # version = item
            # os_release = version

This fixed the issue stated above.

Here is the output of $(get-wmiobject win32_operatingsystem).caption on a Windows 10 Machine

Microsoft Windows 10 Entreprise

There should be an update in the IF statement to add Windows 10 as a OS_RELEASE.

frogunder commented 5 years ago

@saltstack/team-windows any thoughts on this?

morganwillcock commented 5 years ago

The existing code should already match this caption. i.e. splitting the caption will give a "10" and this returns a match:

re.match(r'^(\d+(\.\d+)?)|Thin|Vista$', "10")

I would imagine the problem is where the caption is retrieved. If you print osinfo.Caption what is the value?

alexandrebgallant commented 5 years ago

I added a print(osinfo.Caption) just before the condition,

# This creates the osrelease grain based on the Windows Operating
        # System Product Name. As long as Microsoft maintains a similar format
        # this should be future proof
        version = 'Unknown'
        release = ''
        print(osinfo.caption)
        if 'Server' in osinfo.Caption:
            for item in osinfo.Caption.split(' '):
                # If it's all digits, then it's version
                if re.match(r'\d+', item):
                    version = item
                # If it starts with R and then numbers, it's the release
                # ie: R2
                if re.match(r'^R\d+$', item):
                    release = item
            os_release = '{0}Server{1}'.format(version, release)
        else:
            for item in osinfo.Caption.split(' '):
                # If it's a number, decimal number, Thin or Vista, then it's the
                # version
                if re.match(r'^(\d+(\.\d+)?)|Thin|Vista$', item):
                    version = item
            os_release = version

[INFO    ] Starting up the Salt Minion
[DEBUG   ] AsyncEventPublisher PUB socket URI: 4510
[DEBUG   ] AsyncEventPublisher PULL socket URI: 4511
[INFO    ] Starting pull socket on 4511
[DEBUG   ] SaltEvent PUB socket URI: 4510
[DEBUG   ] SaltEvent PULL socket URI: 4511
[DEBUG   ] Initializing new IPCClient for path: 4510
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
Microsoft Windows 10 Entreprise
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-minion", line 26, in <module>
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-minion", line 26, in <module>
    salt_minion()
  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 191, in salt_minion
    minion.start()
  File "C:\salt\bin\lib\site-packages\salt\cli\daemons.py", line 348, in start
    self.minion.tune_in()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1056, in tune_in
    self._spawn_minions()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1000, in _spawn_minions
    jid_queue=self.jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 972, in _create_minion_object
    jid_queue=jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1126, in __init__
    self.opts['grains'] = salt.loader.grains(opts)
  File "C:\salt\bin\lib\site-packages\salt\loader.py", line 772, in grains
    ret = funcs[key]()
  File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1614, in os_data
    if 'Server' in grains['osrelease']:
TypeError: argument of type 'NoneType' is not iterable    salt_minion()

  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 191, in salt_minion
    minion.start()
  File "C:\salt\bin\lib\site-packages\salt\cli\daemons.py", line 348, in start
    self.minion.tune_in()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1056, in tune_in
    self._spawn_minions()
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1000, in _spawn_minions
    jid_queue=self.jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 972, in _create_minion_object
    jid_queue=jid_queue)
  File "C:\salt\bin\lib\site-packages\salt\minion.py", line 1126, in __init__
    self.opts['grains'] = salt.loader.grains(opts)
  File "C:\salt\bin\lib\site-packages\salt\loader.py", line 772, in grains
    ret = funcs[key]()
  File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1614, in os_data
    if 'Server' in grains['osrelease']:
TypeError: argument of type 'NoneType' is not iterable

The osinfo.Caption returns Microsoft Windows 10 Entreprise

morganwillcock commented 5 years ago

I'm not sure what to suggest, I switched a Windows 10 system to French and gave it a try:

salt-call grains.item osfullname osrelease
local:
    ----------
    osfullname:
        Microsoft Windows 10 Entreprise LTSC
    osrelease:
        10

Potentially it is down to issues with character encodings based on the system language. I found that I couldn't use the Python 2 version at all because the minion crashed trying to parse the time and date information. Which language do you have set as the system language?

alexandrebgallant commented 5 years ago

The language settings in both my test PC are French Canada. I confirm that I installed the salt-minion with python3.

I could try to deploy an english PC and re-test.

morganwillcock commented 5 years ago

I'll try and switch mine to French Canada if you could try English, and hopefully one of us will see a change.

alexandrebgallant commented 5 years ago

With a Windows 10 Enterprise English PC, freshly deployed, the salt-minion works as expected, and doesn't crash.

adminagallant@csvr-saltstack:~$ sudo salt 'ConfMTL*' grains.items
ConfMTL510:
    ----------
    SSDs:
        - \\.\PhysicalDrive0
    biosversion:
        BNKBL357.86A.0069.2018.0920.1250
    cpu_model:
        Intel(R) Core(TM) i5-7260U CPU @ 2.20GHz
    cpuarch:
        AMD64
    disks:
    domain:
    fqdn:
        ConfMTL510
    fqdn_ip4:
        - redacted
    fqdn_ip6:
        - redacted
    fqdns:
    gpus:
    groupname:
    host:
        ConfMTL510
    hwaddr_interfaces:
        ----------
        Intel(R) Ethernet Connection (4) I219-V:
            94:C6:91:AC:40:D0
    id:
        ConfMTL510
    init:
        Windows
    ip4_interfaces:
        ----------
        Intel(R) Ethernet Connection (4) I219-V:
            - redacted
    ip6_interfaces:
        ----------
        Intel(R) Ethernet Connection (4) I219-V:
            - redacted
    ip_interfaces:
        ----------
        Intel(R) Ethernet Connection (4) I219-V:
            - redacted
            - redacted
    ipv4:
        - redacted
    ipv6:
        - redacted
    kernel:
        Windows
    kernelrelease:
        10.0.18362
    kernelversion:
        10.0.18362
    locale_info:
        ----------
        defaultencoding:
            cp1252
        defaultlanguage:
            en_US
        detectedencoding:
            cp1252
        timezone:
            Eastern Daylight Time
    localhost:
        ConfMTL510
    manufacturer:
        Intel(R) Client Systems
    master:
        redacted
    mem_total:
        8080
    motherboard:
        ----------
        productname:
            NUC7i5BNB
        serialnumber:
            GEBN902006PB
    nodename:
        ConfMTL510
    num_cpus:
        4
    num_gpus:
        0
    os:
        Windows
    os_family:
        Windows
    osfinger:
        Windows-10
    osfullname:
        Microsoft Windows 10 Enterprise
    osmanufacturer:
        Microsoft Corporation
    osrelease:
        10
    osrelease_info:
        - 10
    osservicepack:
        None
    osversion:
        10.0.18362
    path:
        C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\salt;C:\windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;c:\salt\bin\lib\site-packages\pywin32_system32;c:\salt\bin\lib\site-packages\pywin32_system32
    pid:
        4264
    productname:
        NUC7i5BNH
    ps:
        tasklist.exe
    pythonexecutable:
        c:\salt\bin\python.exe
    pythonpath:
        - c:\salt\bin\lib\site-packages\git\ext\gitdb
        - c:\salt\bin\Scripts
        - c:\salt\bin\python35.zip
        - c:\salt\bin\DLLs
        - c:\salt\bin\lib
        - c:\salt\bin
        - c:\salt\bin\lib\site-packages
        - c:\salt\bin\lib\site-packages\win32
        - c:\salt\bin\lib\site-packages\win32\lib
        - c:\salt\bin\lib\site-packages\gitdb\ext\smmap
    pythonversion:
        - 3
        - 5
        - 4
        - final
        - 0
    saltpath:
        c:\salt\bin\lib\site-packages\salt
    saltversion:
        2019.2.0
    saltversioninfo:
        - 2019
        - 2
        - 0
        - 0
    serialnumber:
        G6BN90300FPE
    server_id:
        1972647124
    shell:
        C:\windows\system32\cmd.exe
    timezone:
        (UTC-05:00) Eastern Time (US & Canada)
    username:
        CONFMTL510$
    virtual:
        physical
    windowsdomain:
        WORKGROUP
    windowsdomaintype:
        Workgroup
    zfs_feature_flags:
        False
    zfs_support:
        False
    zmqversion:
        4.2.5

morganwillcock commented 5 years ago

Hmm, I'm not sure what makes the difference. I switched mine to 'French (Canada)' and it still worked. If you re-deploy as the original language do you still have the crash?

jiribaloun commented 5 years ago

I did minor change in file c:\salt\bin\lib\site-packages\salt\grains\core.py. line: 1228. At least it's worked for me.

if os_release in ['Unknown']:

replaced by

if 'Unknown' in os_release:

morganwillcock commented 5 years ago

The original purpose is just to check whether the value is 'Unknown' or not. If you print the value of os_release right before this if statement, what is it set to?

phillzen commented 5 years ago

I'm getting the same TypeError on a new Server Core installation (using this ami: https://aws.amazon.com/marketplace/pp/B07R3C9RPW).

(Get-WMIObject win32_operatingsystem).name returns Microsoft Windows Server Datacenter for context.

Before the if statement on line 1255: osinfo.Caption returns Microsoft Windows Server Datacenter os_release returns 10

After the if statement: os_release is UnknownServer

It looks like _clean_value('osrelease', os_release) finds Unknown and returns a value of None. After the grains kv assignment, print(grains['osrelease']) is None.

Leads to: File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1617, in os_data if 'Server' in grains['osrelease']: TypeError: argument of type 'NoneType' is not iterable

xeacott commented 4 years ago

I believe the issue here is that when getting the grains of the machine, osrelease info may have changed moving forward, or whatever the system is needs to check elsewhere if its a 'Server' machine. This could potentially mean using distro to get sys_info, or sys.platform and checking for something else, but its definitely coming from getting system info when collecting grains.

xeacott commented 4 years ago

Hey @phillzen can you send the link to that opensourced gui again?

phillzen commented 4 years ago

Yep it's here https://alcali.dev/ and https://github.com/latenighttales/alcali

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

muddman commented 4 years ago

This also breaks travis ci builds -- test kitchen converge fails when using proxy driver on their windows os. appveyor does work.

$$$$$$   File "C:\salt\bin\lib\site-packages\salt\grains\core.py", line 1662, in os_data
$$$$$$     if 'Server' in grains['osrelease']:
$$$$$$ TypeError: argument of type 'NoneType' is not iterable
The command "powershell -NoProfile -NoLogo -NonInteractive -Command 
'c:\opscode\chefdk\bin\chef.bat exec kitchen test'" exited with 1.
Done. Your build exited with 1.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <>

Travis CI: job log, configuration and code.

AppVeyor: succeeds.

dmurphy18 commented 1 year ago

Closing this since 2019.2.x is long since EOL

saltstack / salt