search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.21k stars 5.48k forks source link

[BUG] KeyError when using s3fs backend on 3006.8 #66473

Open Akrugerus opened 6 months ago

Akrugerus commented 6 months ago

Description After upgrading from 3006.7 to 3006.8 on the salt-master, the following logs are seen in the master log:

2024-05-02 15:38:03,398 [salt.loaded.int.fileserver.s3fs:406 ][DEBUG   ][26069] Refreshing buckets cache file
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:163 ][DEBUG   ][26069] S3 Request: http://my-bucket.s3.us-gov-east-1.amazonaws.com/?marker=
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:164 ][DEBUG   ][26069] S3 Headers::
2024-05-02 15:38:03,399 [salt.loaded.int.utils.s3:165 ][DEBUG   ][26069]     Authorization: AWS4-HMAC-SHA256 Credential=XXX/20240502/us-gov-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=XXX
2024-05-02 15:38:03,400 [urllib3.connectionpool:246 ][DEBUG   ][26069] Starting new HTTP connection (1): my-bucket.s3.us-gov-east-1.amazonaws.com:80
2024-05-02 15:38:03,451 [urllib3.connectionpool:474 ][DEBUG   ][26069] http://my-bucket.s3.us-gov-east-1.amazonaws.com:80 "GET /?marker= HTTP/1.1" 200 None
2024-05-02 15:38:03,452 [salt.loaded.int.utils.s3:228 ][DEBUG   ][26069] S3 Response Status Code: 200
2024-05-02 15:38:03,453 [salt.loaded.int.fileserver.s3fs:636 ][DEBUG   ][26069] Writing buckets cache file
2024-05-02 15:38:03,453 [salt.master      :469 ][ERROR   ][26069] Uncaught exception while updating s3fs fileserver cache
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 467, in _do_update
    update_func(*args)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 125, in update
    metadata = _init()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 371, in _init
    _prune_deleted_files(metadata)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/s3fs.py", line 594, in _prune_deleted_files
    cached_files.add(meta["Key"])
KeyError: 'Key'

Setup

/etc/salt/master.d/fs.conf ```yaml fileserver_backend: - roots - gitfs - s3fs s3.buckets: - my-bucket s3.service_url: s3.us-gov-east-1.amazonaws.com ext_pillar_first: True ext_pillar: - s3: bucket: my-bucket service_url: s3.us-gov-east-1.amazonaws.com prefix: pillar ```

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3006.8 Python Version: Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: 1.6.4 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: 1.12.2 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.16.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: amzn 2 locale: utf-8 machine: x86_64 release: 5.10.149-133.644.amzn2.x86_64 system: Linux version: Amazon Linux 2 ```

Additional details S3 credentials are obtained through the instances IAM role.

I was able to fix the issue with the following patch

diff --git a/salt/fileserver/s3fs.py b/salt/fileserver/s3fs.py
index d3c3d9cd78..cde804beb2 100644
--- a/salt/fileserver/s3fs.py
+++ b/salt/fileserver/s3fs.py
@@ -590,8 +590,9 @@ def _prune_deleted_files(metadata):
                 if os.path.exists(root):
                     roots.add(root)

-            for meta in env_data:
-                cached_files.add(meta["Key"])
+                for meta in env_data:
+                    for obj in meta[bucket]:
+                        cached_files.add(obj["Key"])

     if log.isEnabledFor(logging.DEBUG):
         import pprint
welcome[bot] commented 6 months ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

sudosu101 commented 19 hours ago

seeing the same issue in 3007.1, want to also call out that #60408 is still undocumented and required to pull creds from EC2 IMDS.