search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.17k stars 5.48k forks source link

s3 modules unable to authenticate with Google Cloud Storage s3:// interface #31522

Closed jim-raney-physiq closed 5 years ago

jim-raney-physiq commented 8 years ago

Expected Behavior

To be able to pull files via s3:// protocol from Google Cloud Storage.

Actual Behavior

Authentication fails. It appears that GCS only supports the v2 aws authentication signature for s3:// access, while Salt now only supports v4

Steps to Reproduce Issue

Create a storage bucket in GCS, turn on interoperability, and generate a secret key and access key. In salt minion config set s3.key_id: and s3.access_key: with the generated keys, and set "s3.service_url: storage.googleapis.com". When attempting to access a bucket the salt minion will log "InvalidSecurityThe provided security credentials are not valid.<\Details>Incorrect Authorization header"

Versions Report

Salt Version: Salt: 2015.8.7

Dependency Versions: Jinja2: 2.8 M2Crypto: 0.21.1 Mako: Not Installed PyYAML: 3.11 PyZMQ: 15.2.0 Python: 2.7.5 (default, Nov 20 2015, 02:00:19) RAET: Not Installed Tornado: 4.3 ZMQ: 4.1.2 cffi: 1.5.2 cherrypy: Not Installed dateutil: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed libgit2: Not Installed libnacl: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.7 mysql-python: Not Installed pycparser: 2.14 pycrypto: 2.6.1 pygit2: Not Installed python-gnupg: Not Installed smmap: Not Installed timelib: Not Installed

System Versions: dist: centos 7.2.1511 Core machine: x86_64 release: 3.10.0-327.10.1.el7.x86_64 system: CentOS Linux 7.2.1511 Core

Relevant Logs, Configs, or States

Pre- 2015.5.5 the above steps work without an issue - files can be successfully pulled from GCS via s3://. I'm assuming it's due to https://github.com/saltstack/salt/pull/25420 . We have been working with an older version of Salt and have only recently encountered this issue while trying to upgrade for some other bugfixes. It would be great if Salt would fall back to v2 after a v4 auth attempt fails, or if there was a setting like "s3.sig_version: 2" we could set in the minion config.

jfindlay commented 8 years ago

@jim-raney-physiq, thanks for reporting.

simonyangme commented 8 years ago

Is this still an issue with salt? Is there a recommended workaround?

Is there an ETA for a fix?

jim-raney-physiq commented 8 years ago

This is still a problem as of 2016.3.1. I've been working around it by using a Swift proxy to Google Cloud Storage on the salt master and setting all of the minions to use it (still required hacking up one of the swift modules since someone hard coded a swift version in one instead of using the minion config variable). I'd much rather have S3 v2 back, or a native GCS connector.

jim-raney-physiq commented 8 years ago

We got a note from our account rep that Google updated their storage service to use V4 auth. I haven't been able to get it working yet (get "The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method" on every request). I haven't been able to spend any real time on it.

Capstan commented 8 years ago

OOC, are you signing your User-Agent header?

porunov commented 7 years ago

The same problem I got but with OpenStack Swift with Swift3 module.

Capstan commented 7 years ago

Google Cloud Storage currently has v4 signature support, and the only issue we have seen is with clients who sign their User-Agent header, and an intervening proxy adds extra text to the User-Agent as it gets passed along, which then invalidates said signature. In one case, it was our own proxy, but that has since been addressed. If you are still experiencing authentication failures via v4, I can create you a specific support case where we can compare HTTP request, body of example requests in private.

porunov commented 7 years ago

As I am using my private S3 cloud it isn't convenient for me to use dns-based bucket names. I changed /usr/lib/python2.7/site-packages/salt/utils/s3.py to use path-based bucked names. Here what I have changed:

endpoint = service_url
path = '{0}/{1}'.format(bucket, path)

The problem when I am using salt to get list of files. Here is one of executions:


# salt minion1 s3.get saltbucket top.sls
[DEBUG   ] Configuration file path: /etc/salt/master                                                                                 
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.                                                    
[DEBUG   ] Reading configuration from /etc/salt/master                                                                               
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt1                                                                    
[DEBUG   ] Missing configuration file: /root/.saltrc                                                                                 
[DEBUG   ] RAETEvent Using Jobber Stack at = /var/run/salt/master/master.event543a49f6449a8fe4e3.uxd                                 

[TRACE   ] func get_cli_event_returns()                                                                                              
[DEBUG   ] LazyLoaded local_cache.get_load                                                                                           
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/1d/3c2128dc95bd7466ecba21f9ad728367e221e52536466d9405014512b1fbf7/.minions.p                                                                                                                              
[DEBUG   ] get_iter_returns for jid 20161214230011821533 sent to set([u'minion1']) will timeout at 23:00:16.895679                   
[DEBUG   ] jid 20161214230011821533 return from minion1                                                                              
[DEBUG   ] LazyLoaded nested.output                                                                                                  
[TRACE   ] data = {u'minion1': [odict([(u'Code', u'AuthorizationHeaderMalformed')]), odict([(u'Message', u'The authorization header is malformed; a non-empty region must be provided in the credential.')]), odict([(u'RequestId', u'F3951077BAB13870')]), odict([(u'HostId', u'pSuGv7ivNhI6zE/Ya0GlcX/01X4SVqEahMH29/uIoH5pLarKtGDnBvhZZIrQ1SzTC8bAVmzDqOk=')])]}                                            
minion1:                                                                                                                             
    |_                                                                                                                               
      ----------                                                                                                                     
      Code:                                                                                                                          
          AuthorizationHeaderMalformed                                                                                               
    |_                                                                                                                               
      ----------                                                                                                                     
      Message:
          The authorization header is malformed; a non-empty region must be provided in the credential.
    |_
      ----------
      RequestId:
          F3951077BAB13870
    |_
      ----------
      HostId:
          pSuGv7ivNhI6zE/Ya0GlcX/01X4SVqEahMH29/uIoH5pLarKtGDnBvhZZIrQ1SzTC8bAVmzDqOk=
[DEBUG   ] jid 20161214230011821533 found all minions set([u'minion1'])

But when I apply states they are loaded successfully and accepted by minions. Why is it happens?


# salt '*' state.apply -t 60                          
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt1
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] RAETEvent Using Jobber Stack at = /var/run/salt/master/master.event543a4834e74c2455d8.uxd

[TRACE   ] func get_cli_event_returns()
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/02/ae6e7a912955e38683485633d198335fe2e76e33b8be258a6250935389d14c/.minions.p
[DEBUG   ] get_iter_returns for jid 20161214225220257204 sent to set([u'minion1', u'minion2']) will timeout at 22:53:20.290015
[DEBUG   ] jid 20161214225220257204 return from minion2
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {u'minion2': odict([(u'file_|-/tmp/hello_|-/tmp/hello_|-managed', odict([(u'comment', u'File /tmp/hello updated'), (u'pchanges', odict([(u'diff', u'--- \n+++ \n@@ -1 +1 @@\n-TestOK3!\n+Test4!\n')])), (u'name', u'/tmp/hello'), (u'start_time', u'22:52:29.203129'), (u'result', True), (u'duration', u'10.668 ms'), (u'__run_num__', 0), (u'changes', odict([(u'diff', u'--- \n+++ \n@@ -1 +1 @@\n-TestOK3!\n+Test4!\n')])), (u'__id__', u'/tmp/hello')]))])}
minion2:
----------
          ID: /tmp/hello
    Function: file.managed
      Result: True
     Comment: File /tmp/hello updated
     Started: 22:52:29.203129
    Duration: 10.668 ms
     Changes:   
              ----------                                                                                                                                                                                                                                                       
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -TestOK3!
                  +Test4!

Summary for minion2                                                                                                                                                                                                                                                            
------------                                                                                                                                                                                                                                                                   
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1                                                                                                                                                                                                                                                        
Total run time:  10.668 ms
[DEBUG   ] jid 20161214225220257204 return from minion1
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {u'minion1': odict([(u'file_|-/tmp/hello_|-/tmp/hello_|-managed', odict([(u'comment', u'File /tmp/hello updated'), 556758'), (u'result', True), (u'duration', u'10.146 ms'), (u'__run_num__', 0), (u'changes', odict([(u'diff', u'--- \n+++ \n@@ -1 +1 @
minion1:
----------
          ID: /tmp/hello
    Function: file.managed
      Result: True
     Comment: File /tmp/hello updated
     Started: 22:52:29.556758
    Duration: 10.146 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -TestOK3!
                  +Test4!

Summary for minion1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  10.146 ms
[DEBUG   ] jid 20161214225220257204 found all minions set([u'minion1', u'minion2'])
y33zhang commented 7 years ago

jim-raney-physiq@ - are you still having trouble with the V4 signature? If so, please feel free to contact me at yyzhang@google.com and I can help you take a look at where the signature mismatch is happening.

porunov@ - it's not easy to see what caused the discrepancy from the info you pasted but the reason it failed the first time is that the auth header is missing AWS region (i.e. us-east1 etc). Can you take a look at what is the authheader generated for the request?

jim-raney-physiq commented 7 years ago

I created a trial Google Cloud account and a new bucket, enabled interopability, added a junk file, and tested with s3cmd without a problem. When attempting to get the file via salt (2016.11.1):

root@host:~# salt-call --local s3.get jim-salt-test-01 master.zip.sha256 [DEBUG ] Configuration file path: /etc/salt/minion [WARNING ] Insecure logging configuration detected! Sensitive data may be logged. [DEBUG ] Reading configuration from /etc/salt/minion [DEBUG ] Please install 'virt-what' to improve results of the 'virtual' grain. [DEBUG ] Determining pillar cache [DEBUG ] LazyLoaded jinja.render [DEBUG ] LazyLoaded yaml.render [DEBUG ] compile template: /srv/pillar/top.sls [WARNING ] Template is an empty file: /srv/pillar/top.sls [DEBUG ] LazyLoaded jinja.render [DEBUG ] LazyLoaded yaml.render [DEBUG ] LazyLoaded s3.get [DEBUG ] LazyLoaded config.option [DEBUG ] LazyLoaded s3.query [DEBUG ] Starting new HTTP connection (1): 169.254.169.254 [WARNING ] Failed to get AWS region from instance metadata. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/salt/utils/aws.py", line 552, in get_region_from_metadata proxies={'http': ''}, timeout=AWS_METADATA_TIMEOUT, File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 70, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 479, in send raise ConnectTimeout(e, request=request) ConnectTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/dynamic/instance-identity/document (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f69e6b1e510>, 'Connection to 169.254.169.254 timed out. (connect timeout=3.05)')) [DEBUG ] S3 Request: https://jim-salt-test-01.storage.googleapis.com/master.zip.sha256? [DEBUG ] S3 Headers:: [DEBUG ] Authorization: AWS4-HMAC-SHA256 Credential=GOOGYXUGBMHSDQ6DT34S/20161229/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=5938d7d8635773b87edcdeb48be816a754ddb28ae8842452727beb8afd262cd4 [DEBUG ] Starting new HTTPS connection (1): jim-salt-test-01.storage.googleapis.com [WARNING ] /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:843: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) [DEBUG ] https://jim-salt-test-01.storage.googleapis.com:443 "GET /master.zip.sha256 HTTP/1.1" 403 395 [DEBUG ] Response content: <?xml version='1.0' encoding='UTF-8'?>SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.AWS4-HMAC-SHA256 20161229T181327Z 20161229/us-east-1/s3/aws4_request 88997b60226b7f38c35d2bed6f9cc04359e581a5532aed3a5e534c9c5bcae034 [DEBUG ] S3 Response Status Code: 403 Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 197, in call ret['return'] = func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/salt/modules/s3.py", line 182, in get role_arn=role_arn) File "/usr/lib/python2.7/dist-packages/salt/utils/s3.py", line 225, in query 'Failed s3 operation. {0}: {1}'.format(err_code, err_msg)) CommandExecutionError: Failed s3 operation. SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. Error running 's3.get': Failed s3 operation. SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

/etc/salt/minion contains:

s3.keyid: GOOGYXUGBMHSDQ6DT34S s3.key: s3.service_url: storage.googleapis.com

@Capstan, we're not running through any proxies, the above output came from my workstation which has a direct connection. I've had the same happen on virtual machines in GCE, AWS, wherever, all with direct connections.

@y33zhang, if you're still interested, I can provide the key for the above..

ryandurfey commented 7 years ago

Following up on this thread. We are having similar issues signing in to google cloud with AWS S3 Signature v4. Wondering if anyone has solved this or has additional tips. We have a signature hash that works when logging into AWS S3 using v4, but when sending to Google Cloud (with google key and secret in hash) we get the same error as listed above.

terryhendrix commented 7 years ago

I've got the same problem as @ryandurfey

ryandurfey commented 7 years ago

We got this working recently. One of our engineers figured out our issue. We were using the wrong hostname. You have to use storage.googleapis.com/bucket/ rather than bucket.storage.googleapis.com

Capstan commented 7 years ago

Well, that's unfortunate. I should think both would work. I'll prod at the dev in charge of those tests to validate.

ryandurfey commented 7 years ago

@Capstan Any luck on validation? Our business case depends upon failing over between two regional origins on google cloud and this requires us to use the explicit domain name (vs. generic domain/bucket option).

Capstan commented 7 years ago

Yes, finally. We have a fix in the queue, though it'll take some time to show up in production.

ryandurfey commented 7 years ago

Thanks @Capstan! I wasn't sure how current this thread was so I opened a google issue https://issuetracker.google.com/issues/62410187 which references back to this. Apologies if this was just duplication.

Capstan commented 7 years ago

No worries. Internally we live and breathe on the issue tracker, so I'd already made a private one; I've marked it blocking yours and this.

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.