metadata grains fails on gzipped user-data

vanveele commented 3 years ago

Description of Issue

With metadata_server_grains: true, minion fails to fetch metadata when instance user-data is binary or gzip data. Call to salt.utils.http.query fails to decode body as unicode:

[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <function metadata at 0x7ff527139730>, error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 853, in grains
    ret = funcs[key](**kwargs)
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 89, in metadata
    return _search()
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 66, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 50, in _search
    linedata = http.query(os.path.join(HOST, prefix), headers=True)
  File "/usr/lib/python3.6/site-packages/salt/utils/http.py", line 659, in query
    result_text = result_text.decode("utf-8")

A possible fix could catch the UnicodeDecodeError and return. eg.

def _search(prefix="latest/"):
    """
    Recursively look up all grains in the metadata server
    """
    ret = {}
    try:
      linedata = http.query(os.path.join(HOST, prefix), headers=True)
    except UnicodeDecodeError:
      return ret

excerpt of DEBUG log:

# salt-call -ldebug grains.get meta-data
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/metadata_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/metadata_server.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minion
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
{...}
[DEBUG   ] Requesting URL http://169.254.169.254/latest/meta-data/services/ using GET method
[DEBUG   ] Using backend: tornado
[DEBUG   ] Response Status Code: 200
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/metadata_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/metadata_server.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minion
[DEBUG   ] Requesting URL http://169.254.169.254/latest/meta-data/services/domain using GET method
[DEBUG   ] Using backend: tornado
[DEBUG   ] Response Status Code: 200
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/metadata_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/metadata_server.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minion
[DEBUG   ] Requesting URL http://169.254.169.254/latest/meta-data/services/partition using GET method
[DEBUG   ] Using backend: tornado
[DEBUG   ] Response Status Code: 200
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/metadata_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/metadata_server.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minion
[DEBUG   ] Requesting URL http://169.254.169.254/latest/user-data/ using GET method
[DEBUG   ] Using backend: tornado
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <function metadata at 0x7f27c527d730>, error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 853, in grains
    ret = funcs[key](**kwargs)
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 89, in metadata
    return _search()
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 66, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 50, in _search
    linedata = http.query(os.path.join(HOST, prefix), headers=True)
  File "/usr/lib/python3.6/site-packages/salt/utils/http.py", line 659, in query
    result_text = result_text.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
[DEBUG   ] LazyLoaded zfs.is_supported

Setup

launch ec2 with binary / gzip user-data
install salt-minion
enable metadata_server_grains: true in /etc/salt/minion

Steps to Reproduce Issue

salt-call grains.get meta-data

also reproducible from shell:

# python3 -c 'from salt.grains.metadata import metadata; metadata();'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 89, in metadata
    return _search()
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 66, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/usr/lib/python3.6/site-packages/salt/grains/metadata.py", line 50, in _search
    linedata = http.query(os.path.join(HOST, prefix), headers=True)
  File "/usr/lib/python3.6/site-packages/salt/utils/http.py", line 659, in query
    result_text = result_text.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Versions Report

Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: 1.14.3
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: 2.6.1
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: ANSI_X3.4-1968
       machine: x86_64
       release: 3.10.0-1127.19.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

welcome[bot] commented 3 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at core@saltstack.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

max-arnold commented 8 months ago

I believe that parsing of the user-data should be avoided. Another case when it fails is when it contains a cloud-config yaml document (the parser tries to iterate over it and fetch each line as another metadata): https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-cloud-init

It is worth skipping the user-data:


--- metadata.orig.py    2024-01-15 03:12:26.307231394 +0000
+++ metadata.py 2024-01-15 02:57:37.087629576 +0000
@@ -55,6 +55,8 @@
     ):
         return body
     for line in body.split("\n"):
+        if line == "user-data":
+            continue
         if line.endswith("/"):
             ret[line[:-1]] = _search(prefix=os.path.join(prefix, line))
         elif prefix == "latest/":```

saltstack / salt