search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.11k stars 5.47k forks source link

Salt can't get states from S3 storage #38215

Closed porunov closed 7 years ago

porunov commented 7 years ago

Description of Issue/Question

Hello, I have a problem with using s3 as a fileserver_backend. When I am trying to update states then it just getting stuck and doing nothing.

Why states can not be downloaded?

Sincerely, Alexandr

Setup

Here is my master configuration:

...
fileserver_backend:
  - s3fs
s3.keyid: <ec2_id>
s3.key: <ec2_key>
s3.service_url: 192.168.0.42:8080
s3.verify_ssl: False
s3.buckets:
  base:
    - salt
...

I have next files in the 'salt' bucket:

top.sls
webserver/init.sls
webserver/httpd.conf
webserver/index.html

I am using my own s3 backend (OpenStack Swift with Swift3 module for S3 api).

Steps to Reproduce Issue

When I run:

salt '*' state.apply -t 60

Nothing happens. It just gets stuck and then shows an error that 0 minions sent responses.

Here is errors from /var/log/salt/master:

ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
2016-12-12 23:54:56,512 [salt.utils.aws   ][WARNING ][6754] Failed to get AWS region from instance metadata.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/aws.py", line 552, in get_region_from_metadata
    proxies={'http': ''}, timeout=AWS_METADATA_TIMEOUT,
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(113, 'No route to host'))
2016-12-12 23:55:22,305 [salt.daemons.masterapi][ERROR   ][6753] Exception ('Connection aborted.', gaierror(-2, 'Name or service not known')) occurred in file server update
2016-12-12 23:56:23,127 [salt.daemons.masterapi][ERROR   ][6753] Exception ('Connection aborted.', gaierror(-2, 'Name or service not known')) occurred in file server update
2016-12-13 02:19:22,078 [py.warnings      ][WARNING ][2014] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:22,929 [py.warnings      ][WARNING ][2020] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:22,935 [py.warnings      ][WARNING ][2022] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:22,947 [py.warnings      ][WARNING ][2015] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:22,959 [py.warnings      ][WARNING ][2018] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:22,968 [py.warnings      ][WARNING ][2017] /usr/lib/python2.7/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.

2016-12-13 02:19:25,517 [salt.utils.aws   ][WARNING ][2014] Failed to get AWS region from instance metadata.

Versions Report

Salt Version: Salt: 2016.3.3

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: 1.6.5 Jinja2: 2.7.2 libgit2: Not Installed libnacl: 1.5.0 M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pygit2: Not Installed Python: 2.7.5 (default, Sep 15 2016, 22:37:39) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: 0.6.5 smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.2.1511 Core machine: x86_64 release: 3.10.0-327.4.5.el7.x86_64 system: Linux version: CentOS Linux 7.2.1511 Core

Ch3LL commented 7 years ago

@porunov this looks like some possible network issues. In the code here its trying to get to this url: http://169.254.169.254/latest/dynamic/instance-identity/document which if i recall correctly that 169 address is used to collect metadata. I'm guessing for some reaosn you can not query that adddress.

lorengordon commented 7 years ago

Probably the s3 module is depending a bit too heavily on AWS specifics, which makes it difficult for use cases like this with a custom s3 backend...

Digging around a bit more, seems that you could perhaps avoid that call to get_region_from_metadata() by setting s3.location in your master config. That location should get passed to utils.s3.query(), which passes it to utils.aws.sig4(), which uses the location to create the signing key. I have no idea show Swift3 works, or if it support sigv4 signing, but hopefully this will give you enough clues to get it working!

porunov commented 7 years ago

But I have fully compatible S3 storage. Both signature v2 and signature v4 work. I am using official Java AWS S3 client to work with my storage and it works pretty well.

I tried to set s3.location in the salt master but it didn't have an effect.

Also about http://169.254.169.254/latest/dynamic/instance-identity/document I really can not call it. But I am not sure why. I have a normal internet connection.

Sincerely, Alexandr

porunov commented 7 years ago

I found out that it sends a https connection not http. I have set s3.verify_ssl: False but it still sends https requests. Is it possible to change?

porunov commented 7 years ago

Also the real problem for me is that it sends messages to <bucket>.<mydomain>.com. It would be great if it was possible to send requests like this: <mydomain>.com/<bucket>

lorengordon commented 7 years ago

I'm really not familiar with Swift3, so I'm afraid I won't be any further help. Would you maybe be able to test using AWS S3 briefly, to confirm your configuration works there?

Also the real problem for me is that it sends messages to ..com. It would be great if it was possible to send requests like this: .com/

If that is root of the problem, you may want to change this to a feature request to support path-based bucket URIs. You can see here in the code that s3.query is using the combination of your s3.service_url and the bucket name to construct the bucket.service_url endpoint.

porunov commented 7 years ago

Thank you for your help. I tried it but it says that signature doesn't work...

There is a cool cli client called s3curl.pl and it works nicely but salt can't retrieve a list of objects... Here is what I do: ./s3curl.pl --id <my_id> --key <my_key> https://mydomain.com/saltbucket

It shows me an XML with objects which is saved in a storage:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Name>saltbucket</Name>
   <Prefix />
   <Marker />
   <MaxKeys>1000</MaxKeys>
   <IsTruncated>false</IsTruncated>
   <Contents>
      <Key>base/top.sls</Key>
      <LastModified>2016-12-13T21:30:33.528Z</LastModified>
      <ETag>"d6d9711ff5e2240326a459f83e9b2aeb"</ETag>
      <Size>30</Size>
      <Owner>
         <ID>service:swift</ID>
         <DisplayName>service:swift</DisplayName>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
   </Contents>
   <Contents>
      <Key>base/webserver/init.sls</Key>
      <LastModified>2016-12-13T21:36:33.831Z</LastModified>
      <ETag>"ec6f47c6da52d39addfbc5f0e72bd252"</ETag>
      <Size>72</Size>
      <Owner>
         <ID>service:swift</ID>
         <DisplayName>service:swift</DisplayName>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
   </Contents>
   <Contents>
      <Key>top.sls</Key>
      <LastModified>2016-12-13T21:30:16.205Z</LastModified>
      <ETag>"d6d9711ff5e2240326a459f83e9b2aeb"</ETag>
      <Size>30</Size>
      <Owner>
         <ID>service:swift</ID>
         <DisplayName>service:swift</DisplayName>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
   </Contents>
   <Contents>
      <Key>webserver/init.sls</Key>
      <LastModified>2016-12-13T21:36:48.143Z</LastModified>
      <ETag>"ec6f47c6da52d39addfbc5f0e72bd252"</ETag>
      <Size>72</Size>
      <Owner>
         <ID>service:swift</ID>
         <DisplayName>service:swift</DisplayName>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
   </Contents>
</ListBucketResult>

I have changed that code which you showed me from: endpoint = '{0}.{1}'.format(bucket, service_url) to endpoint = '{1}/{0}'.format(bucket, service_url)

But when I am using salt to show a list of files in the bucket it shows that signature isn't correct. I am using: # salt minion1 s3.get saltbucket top.sls

It shows:

minion1:
    |_
      ----------
      Code:
          AuthorizationHeaderMalformed
    |_
      ----------
      Message:
          The authorization header is malformed; a non-empty region must be provided in the credential.
    |_
      ----------
      RequestId:
          4E81D1E9C24108BE
    |_
      ----------
      HostId:
          rHFLVzoH4Y1FfEpHMAAsl6rlf9jyv2U2eZVGvy8kjne0zwFQ5AzDIgeO4iGu+nJoJ2UNbc6poF0=

In log files I see next problems:

...
2016-12-14 17:59:37,169 [salt.utils.lazy  ][DEBUG   ][31733] LazyLoaded local_cache.clean_old_jobs
2016-12-14 17:59:37,171 [salt.fileserver  ][DEBUG   ][31733] Updating s3fs fileserver cache
2016-12-14 17:59:37,171 [salt.loaded.int.fileserver.s3fs][DEBUG   ][31733] Refreshing buckets cache file
2016-12-14 17:59:37,171 [salt.utils.s3    ][DEBUG   ][31733] S3 Request: https://jblur.com:443/saltbucket/?
2016-12-14 17:59:37,171 [salt.utils.s3    ][DEBUG   ][31733] S3 Headers::
2016-12-14 17:59:37,172 [salt.utils.s3    ][DEBUG   ][31733]     Authorization: AWS4-HMAC-SHA256 Credential=96261b54efcf4ca98d00822d6e60ddb2/20161214/US/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=908a0e7f21a2e4cf75d7421acde0208703c67b6a854651667c3cfb0cdd5a278a
2016-12-14 17:59:37,173 [requests.packages.urllib3.connectionpool][INFO    ][31733] Starting new HTTPS connection (1): jblur.com
2016-12-14 17:59:37,312 [requests.packages.urllib3.connectionpool][DEBUG   ][31733] "GET /saltbucket/ HTTP/1.1" 403 None
2016-12-14 17:59:37,313 [salt.utils.s3    ][DEBUG   ][31733]     Response content: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>tx7b19f6d79bb4435d9ce1b-0058516c69</RequestId></Error>
2016-12-14 17:59:37,313 [salt.utils.s3    ][DEBUG   ][31733] S3 Response Status Code: 403
2016-12-14 17:59:37,314 [salt.loaded.int.fileserver.s3fs][WARNING ][31733] 'The request signature we calculated does not match the signature you provided. Check your key and signing method.' response for bucket 'saltbucket'
2016-12-14 17:59:37,314 [salt.loaded.int.fileserver.s3fs][DEBUG   ][31733] Writing buckets cache file
2016-12-14 17:59:37,314 [salt.loaded.int.fileserver.s3fs][INFO    ][31733] Syncing local cache from S3...
2016-12-14 17:59:37,315 [salt.loaded.int.fileserver.s3fs][INFO    ][31733] Sync local cache from S3 completed.
...

Here is my /etc/salt/master configuration file:

fileserver_backend:
  - s3fs
s3.keyid: <id>
s3.key: <key>
s3.location: US
s3.service_url: mydomain.com
s3.buckets:
  - saltbucket
s3.verify_ssl: False

Do you know why the signature doesn't work?

porunov commented 7 years ago

Also I tried to change s3.py like this:

#    if bucket:
#        endpoint = '{1}'.format(bucket, service_url)
#    else:

    endpoint = service_url
    path = '{0}/{1}'.format(bucket, path)

Got the same signature error

lorengordon commented 7 years ago

Really not sure. Maybe start sprinkling log statements throughout to try to debug that the values being passed around are what you'd expect?

porunov commented 7 years ago

I am trying but without success. Also I don't know Python.. So it's complicated for me to figure out where is the problem

porunov commented 7 years ago

Issue was resolved by adding http support and path style support:

38295