s3_bucket -- DigitialOcean Spaces compatibility

eeshugerman commented 3 years ago

SUMMARY

I'm upgrading from Ansible v2.8 to v4.1. With v2.8, I was able to use s3_bucket with DO Spaces -- I just needed to set requester_pays: null. Now, with v4.1, it seems there's no way to prevent the module's get_bucket_encryption call, and DO Spaces doesn't implement encryption stuff, so the task fails. I tried all of the permitted values for encryption ('none', 'AES256' 'aws:kms'), as well as not setting it at all -- it made no difference.

See https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_bucket.py#L420-L426. My naive assessment is that the get_bucket_encryption call (and its try/except block) would need to be moved into the if encryption is not None block.

ISSUE TYPE

Bug Report

COMPONENT NAME

s3_bucket

ANSIBLE VERSION

ansible [core 2.11.1] 
  config file = /home/elliott/devel/PikaNet/ansible.cfg
  configured module search path = ['/home/elliott/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/elliott/.virtualenvs/ansible-4/lib/python3.9/site-packages/ansible
  ansible collection location = /home/elliott/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/elliott/.virtualenvs/ansible-4/bin/ansible
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True

CONFIGURATION

ANSIBLE_FORCE_COLOR(/home/elliott/devel/PikaNet/ansible.cfg) = True
DEFAULT_STDOUT_CALLBACK(/home/elliott/devel/PikaNet/ansible.cfg) = yaml
DEFAULT_VAULT_PASSWORD_FILE(/home/elliott/devel/PikaNet/ansible.cfg) = /home/elliott/devel/PikaNet/.vault_pass
INTERPRETER_PYTHON(/home/elliott/devel/PikaNet/ansible.cfg) = auto

OS / ENVIRONMENT

Ubuntu 18.04

STEPS TO REPRODUCE

amazon.aws.s3_bucket:
  name: 'my-bucket'
  state: present
  aws_access_key: '{{ digitalocean_spaces.key }}'
  aws_secret_key: '{{ digitalocean_spaces.secret }}'
  s3_url: 'https://sfo2.digitaloceanspaces.com'

EXPECTED RESULTS

The task succeeds.

ACTUAL RESULTS

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 388, in create_or_update_bucket
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
    raise e
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
    return f(*args, **kwargs)
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 522, in get_bucket_encryption
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
fatal: [default]: FAILED! => changed=false 
  boto3_version: 1.4.2
  botocore_version: 1.16.19
  error:
    code: NotImplemented
    message: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
  invocation:
    module_args:
      aws_access_key: <REDACTED>
      aws_ca_bundle: null
      aws_config: null
      aws_secret_key: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      ceph: false
      debug_botocore_endpoint_logs: false
      delete_public_access: false
      ec2_url: null
      encryption: null
      encryption_key_id: null
      force: false
      name: my-bucket
      policy: null
      profile: null
      public_access: null
      purge_tags: true
      region: null
      requester_pays: null
      s3_url: https://sfo2.digitaloceanspaces.com
      security_token: null
      state: present
      tags: null
      validate_certs: true
      versioning: null
  msg: 'Failed to get bucket encryption: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility'
  response_metadata:
    http_headers:
      content-length: '268'
      content-type: application/xml
      strict-transport-security: max-age=15552000; includeSubDomains; preload
      x-amz-error-code: NotImplemented
      x-amz-error-message: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
      x-do-spaces-error: unsupported_operation_s3:GetBucketEncryptionConfiguration
    http_status_code: 501
    retry_attempts: 0

ansibullbot commented 3 years ago

Files identified in the description: None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

eeshugerman commented 3 years ago

!component =plugins/modules/s3_bucket.py

ansibullbot commented 3 years ago

Files identified in the description:

[plugins/modules/s3_bucket.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 3 years ago

cc @abikouo @jillr @s-hertel @tremble @wimnat click here for bot help

tremble commented 3 years ago

The fix for this issue should be available when we release version 2.0.0 of the amazon.aws collection.

eeshugerman commented 3 years ago

thanks @tremble!

ansible-collections / amazon.aws