mine.get unable to target with mine cache data.

[kschafer2598]

Description of Issue

Hi,

Trying to do mine.get and it is not returning results when using grain targeting. This is a multimaster setup with etcd for the cache.

Targeting with glob works; Targeting with grains or compound does not work.

Works: salt-call mine.get '*' grains.item

Does not work: salt-call mine.get 'cpuarch:x86_64' grains.item grain salt-call mine.get 'G@cpuarch:x86_64' grains.item compound

Setup

Minion mine config:

mine_interval: 5
mine_functions:
  grains.item: [fqdn, ipv4, fqdn_ip4]

Steps to Reproduce Issue

salt-call mine.get 'cpuarch:x86_64' grains.item grain

Versions Report

Minion:

Salt Version: Salt: 2019.2.2

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.5 (default, Aug 7 2019, 00:51:29) python-gnupg: Not Installed PyYAML: 3.10 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.7.1908 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1062.1.1.el7.x86_64 system: Linux version: CentOS Linux 7.7.1908 Core

Master:

Salt Version: Salt: 2019.2.2

Dependency Versions: cffi: Not Installed cherrypy: 17.4.2 dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.17 (default, Nov 7 2019, 10:07:09) python-gnupg: 0.4.1 PyYAML: 3.12 PyZMQ: 16.0.2 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.5

System Versions: dist: Ubuntu 18.04 bionic locale: UTF-8 machine: x86_64 release: 4.15.0-76-generic system: Linux version: Ubuntu 18.04 bionic

[DmitryKuzmenko]

2020-02-17 13:54:37 dimm@kuzmenko /home/dimm/projects/salt/git/salt (⌐■_■) (HEAD detached at v2019.2.2)
(venv-2.7) $ ./scripts/salt -l critical '*' mine.get 'os:Arch' internal_ip grain
alpha:
    ----------
    alpha:
        - 192.168.48.28
2020-02-17 13:54:41 dimm@kuzmenko /home/dimm/projects/salt/git/salt (⌐■_■) (HEAD detached at v2019.2.2)
(venv-2.7) $ ./scripts/salt-call -l critical mine.get 'os:Arch' internal_ip grain
local:
    ----------
    alpha:
        - 192.168.48.28
2020-02-17 13:54:46 dimm@kuzmenko /home/dimm/projects/salt/git/salt (⌐■_■) (HEAD detached at v2019.2.2)
(venv-2.7) $ ./scripts/salt-call -l critical  mine.get 'G@os:Arch' internal_ip compound
local:
    ----------
    alpha:
        - 192.168.48.28

[DmitryKuzmenko]

Could you please provide more details about your configuration? Sanitized minion config would be perfect.

[kschafer2598]

Here's the minion config

master:
- salt-master-01
- salt-master-02

user: root

sudo_user: root

environment: prod

log_level_logfile: info

mine_interval: 5
mine_functions:
  grains.item: [fqdn, ipv4, fqdn_ip4, roles, custom_grain]

[kschafer2598]

I am also unable to get mine data using templates:

{% for host, data in salt['mine.get']('G@kernel:Linux', 'grains.item', 'compound').items() -%}
{{ data["fqdn"] }}
{% endfor -%}

[kschafer2598]

I haven't seen any updates; Is this the right place to report an issue like this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[kschafer2598]

Please help; let me know what else is needed to move forward

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[DmitryKuzmenko]

@kschafer2598 sorry missed that. I'll follow up today.

[DmitryKuzmenko]

Still cannot reproduce the issue but I'm working on that. Will try more tomorrow.

[DmitryKuzmenko]

No luck. @team-core could someone try to reproduce this?

[OrangeDog]

Does it work with kwargs, tgt_type=grain, instead of relying on the order?

[kschafer2598]

Just tried it, same result

root@salthost-01:~# salt-call mine.get 'os:Ubuntu' grains.item tgt_type=grain
local:
    ----------

[kschafer2598]

I've upgraded minions/masters to 2019.2.5 and have the same issue.

What else can I try here?

[DmitryKuzmenko]

@kschafer2598 I'm getting the empty results if I'm removing cache and run master and never run minion, using the only salt-call command. After I'm starting the minion once everything works good ever if I'm stopping it.

[DmitryKuzmenko]

Could you also please check is this the same on both masters?

[github-abcde]

Could you try the following:

• verify on all related minions that salt $minion config.get mine_functions produces the correct result.
• Flush all previous mine data using salt $minion mine.flush
• Forcing an update of the mine data using salt $minion mine.update
• Optionally: verify the targeting grain using salt $minion grains.get cpuarch
• Then query the salt-mine, using salt $minion mine.get 'cpuarch:x86_64' grains.item grain Of course these could also all be done locally using salt-call instead of salt $minion. I'm interested in where things go awry.

[kschafer2598]

Thanks, everything works until the end, where it returns nothing for the query.

root@salthost-01:~# salt-call grains.get os
local:
    Ubuntu

root@salthost-01:~# salt-call mine.get 'os:Ubuntu' grains.item grain
local:
    ----------

[ikogan]

I'm also experiencing what I believe to be this behavior, I cannot target with grains but using individual minion IDs works just fine.

Salt Version:
           Salt: 3000.4

Dependency Versions:
           cffi: 1.14.3
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: 1.1.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: 3.9.9
         pygit2: 1.4.0
         Python: 3.7.9 (default, Oct 22 2020, 05:22:26)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist:   
         locale: UTF-8
        machine: x86_64
        release: 5.4.0-72-generic
         system: Linux
        version: Not Installed

Minion Version is 3001.7.

[github-abcde]

@ikogan Could you post the commands that you tried and the output they gave you? That would give us more insight into what exactly you're trying (and what salt is doing or not doing for you) and will help us formulate useful followup questions.

[ikogan]

It appears that in attempting to reproduce this behavior, I've run into another issue where I cannot target minions by a list value. As I understand it, the following should work:

# salt 'vpn.*' grains.get roles
/usr/local/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
vpn.private.local:
    - vpn-server
# salt -G 'roles:vpn-server' test.ping
/usr/local/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received
command terminated with exit code 2

[github-abcde]

Please create a separate issue for this, as the "other issue" you have described is not related to the salt mine. (And as always, check if there is a similar issue about the specific behavior you're seeing first).

[kschafer2598]

Sorry for the long delay, I got around to installing new masters w/ 3003.1 packages. The results from the steps mentioned above were the same however - the actual salt mine query returns nothing.

verify on all related minions that salt $minion config.get mine_functions produces the correct result.
Flush all previous mine data using salt $minion mine.flush
Forcing an update of the mine data using salt $minion mine.update
Optionally: verify the targeting grain using salt $minion grains.get cpuarch
Then query the salt-mine, using salt $minion mine.get 'cpuarch:x86_64' grains.item grain
Of course these could also all be done locally using salt-call instead of salt $minion.
I'm interested in where things go awry.

At this point I can only assume something is wrong with my configuration - are there any examples of working salt mine configurations I can go by?

[kschafer2598]

I was doing more testing and came across an etcd error that occurs at the time of the mine.get; It appears to be looking for a path that doesn't exist, /data/data, which seems like a typo. I can't see anywhere in configs where it is set.

2021-08-03 18:47:54,321 [etcd.client      :126 ][ERROR   ][29369] Could not discover the etcd hosts from None: None of DNS query names exist: _etcd._tcp.None., _etcd._tcp.None.openstacklocal.
2021-08-03 18:47:54,321 [urllib3.connectionpool:208 ][DEBUG   ][29369] Starting new HTTP connection (1): 127.0.0.1
2021-08-03 18:47:54,322 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/machines HTTP/1.1" 200 21
2021-08-03 18:47:54,323 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
2021-08-03 18:47:54,324 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 446
2021-08-03 18:47:54,324 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 446
2021-08-03 18:47:54,325 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions/testhost.blah.com HTTP/1.1" 200 None
2021-08-03 18:47:54,326 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions/saltmaster-01.blah.com HTTP/1.1" 200 None
2021-08-03 18:47:54,327 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions/saltmaster-02.blah.com HTTP/1.1" 200 None
2021-08-03 18:47:54,328 [urllib3.connectionpool:396 ][DEBUG   ][29369] http://127.0.0.1:2379 "GET /v2/keys/salt_cache/minions/testhost.blah.com/data/data HTTP/1.1" 400 129
2021-08-03 18:47:54,328 [salt.utils.minions:729 ][ERROR   ][29369] Failed matching available minions with grain pattern: fqdn:testhost.blah.com
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/cache/etcd_cache.py", line 148, in fetch
    value = client.read(etcd_key).value
  File "/usr/local/lib/python3.6/dist-packages/etcd/client.py", line 597, in read
    timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/etcd/client.py", line 907, in wrapper
    return self._handle_server_response(response)
  File "/usr/local/lib/python3.6/dist-packages/etcd/client.py", line 987, in _handle_server_response
    etcd.EtcdError.handle(r)
  File "/usr/local/lib/python3.6/dist-packages/etcd/__init__.py", line 306, in handle
    raise exc(msg, payload)
etcd.EtcdNotDir: Not a directory : /salt_cache/minions/testhost.blah.com/data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/minions.py", line 714, in check_minions
    _res = check_func(expr, delimiter, greedy)
  File "/usr/lib/python3/dist-packages/salt/utils/minions.py", line 349, in _check_grain_minions
    return self._check_cache_minions(expr, delimiter, greedy, "grains")
  File "/usr/lib/python3/dist-packages/salt/utils/minions.py", line 328, in _check_cache_minions
    mdata = self.cache.fetch("minions/{}".format(id_), "data")
  File "/usr/lib/python3/dist-packages/salt/cache/__init__.py", line 182, in fetch
    return self.modules[fun](bank, key, **self._kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1241, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2274, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2289, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cache/etcd_cache.py", line 154, in fetch
    "There was an error reading the key, {}: {}".format(etcd_key, exc)
salt.exceptions.SaltCacheError: There was an error reading the key, /salt_cache/minions/testhost.blah.com/data/data: Not a directory : /salt_cache/minions/testhost.blah.com/data

Also: Mine data can be found in etcd via curl: curl localhost:2379/v2/keys/salt_cache/minions/testhost.blah.com/mine | jq ".node.value" | sed 's/"//g' | base64 -d

[denis-khalturin-incountry]

I have the same problem with the enabled caching setting - "cache: redis". When the "cache" parameter uses the default parameter (localfs), then mine.get with grain targeting works.

@kschafer2598 how was it for you? @DmitryKuzmenko jfyi

[keslerm]

We hit this exact same issue using the redis cache.

We are using this in a state file to pull a list of ip addresses.

We were originally trying to target via pillar, but we tested grains as well and it was the same issue.

Switching back to localfs cache fixes it

Targeting by pillar/grain works everywhere else I tested, it's only using the mine.get function

[whytewolf]

So, This may or may not be fixed in the upcoming 3005. I know @waynew has been doing a ton of work in the data cache space. I would love some one who is running into the issue and knows it inside and out to test with the current master branch of salt to see if it is fixed.

@waynew can you post your PR numbers for the cache fixes you have put in?

[kschafer2598]

Glad to hear this may be getting some attention, when will 3005 be released?

[keslerm]

In our test setup this seems to be resolved