HTTP 400 Status Code and Error when listing buckets

vmorris commented 3 years ago

When attempting to list all the buckets in my COS instance (there is only one), I am getting an error:

ibm_botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the ListBuckets operation: Invalid Argument

$ pip list | grep cos
ibm-cos-sdk            2.7.0     
ibm-cos-sdk-core       2.7.0     
ibm-cos-sdk-s3transfer 2.7.0

I instantiate a client and attempt to use it:

client = ibm_boto3.resource(
    "s3",
    ibm_api_key_id=config["api_key"],
    ibm_service_instance_id=config["instance_crn"],
    ibm_auth_endpoint=config["auth_endpoint"],
    endpoint_url=config["endpoint"],
    config=Config(signature_version="oauth"),
)
buckets = client.buckets.all()
for bucket in buckets:
    print(bucket.name)

Attempting to list the objects inside the known bucket does work. Example (working) snippet:

objects = client.Bucket("mybucket").objects.all()
for obj in objects:
    print(f"{obj.key} ({obj.size} bytes)")

Full traceback:

Traceback (most recent call last):
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/bin/cos", line 11, in <module>
    load_entry_point('cos', 'console_scripts', 'cos')()
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/cos.py", line 67, in buckets
    for bucket in buckets:
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/ibm_boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/ibm_boto3/resources/collection.py", line 161, in pages
    pages = [getattr(client, self._py_operation_name)(**params)]
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/ibm_botocore/client.py", line 380, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/ibm_botocore/client.py", line 752, in _make_api_call
    if http.status_code >= 300:
ibm_botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the ListBuckets operation: Invalid Argument

vmorris commented 3 years ago

Setting a breakpoint with pdb and inspecting the HTTP response, I find the 400 code:

> /home/vmorris/git/github.ibm.com/zaas/scaas/tools/cos_cli/venv/lib64/python3.8/site-packages/ibm_botocore/client.py(752)_make_api_call()
-> if http.status_code >= 300:
(Pdb) http
<ibm_botocore.awsrequest.AWSResponse object at 0x7fdc9e1019a0>
(Pdb) http.text
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Error><Code>InvalidArgument</Code><Message>Invalid Argument</Message><RequestId>548264d0-9470-42dc-9dc4-ef0ce3335ad5</RequestId><httpStatusCode>400</httpStatusCode></Error>'

vmorris commented 3 years ago

And, just a bit more info about the request:

(Pdb) request_context
{'client_region': 'us-east-1', 'client_config': <ibm_botocore.config.Config object at 0x7fdc9e1d0400>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}
(Pdb) request_dict
{'url_path': '/', 'query_string': {}, 'method': 'GET', 'headers': {'ibm-service-instance-id': 'crn:v1:bluemix:public:cloud-object-storage:global:a/REDACTED:REDACTED:bucket:myjunk', 'User-Agent': 'ibm-cos-sdk-python/2.7.0 Python/3.8.5 Linux/5.8.14-200.fc32.x86_64 ibm-cos-sdk-python-core/2.7.0 Resource'}, 'body': b'', 'url': 'https://s3.us-south.cloud-object-storage.appdomain.cloud/', 'context': {'client_region': 'us-east-1', 'client_config': <ibm_botocore.config.Config object at 0x7fdc9e1d0400>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}

IBMeric commented 3 years ago

Thank you for your question. Your ibm_service_instance_id is incorrect. From the log, you are using the CRN of the bucket, not of the service instance. If you remove :bucket:myjunk that should work. You also should be able to use your service ID directly (the GUID segment). An example of service credentials can be found in the documentation here: https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials.

If this solves your problem, please mark the ticket closed.

vmorris commented 3 years ago

Thank you @IBMeric , that was exactly the problem. Cheers.

IBM / ibm-cos-sdk-python

HTTP 400 Status Code and Error when listing buckets #38