Tencent cloud cos(s3) upload failed

[dispensable]

Project board link

I am using tencent cloud(cloud.tencent.com) cos (s3 compat service) for cassandra backup. It failed with uploadId None.

After some investigations, I find libcloud will use default aws NAMESPACE for xml response parse. But tencent cloud has their own xmlns field or in same case there is just no xmlns field exists. I can't find ns settings in ini file. So I fixed ns setting with this patch. But I am wondering if there is a option to setting xmlns field ? Or should i open a issue for libcloud lib and tencent maby ?

--- /usr/local/lib/python3.9/dist-packages/libcloud/storage/drivers/s3.py   2023-07-28 15:05:18.919248455 +0800
+++ s3_fixed.py 2023-07-28 15:04:26.435089224 +0800
@@ -108,7 +108,7 @@
 }

 API_VERSION = '2006-03-01'
-NAMESPACE = 'http://s3.amazonaws.com/doc/%s/' % (API_VERSION)
+NAMESPACE = "http://www.qcloud.com/document/product/436/7751"

 # AWS multi-part chunks must be minimum 5MB
 CHUNK_SIZE = 5 * 1024 * 1024
@@ -576,8 +576,13 @@
             raise LibcloudError('Error initiating multipart upload',
                                 driver=self)

-        return findtext(element=response.object, xpath='UploadId',
-                        namespace=self.namespace)
+        upload_id = findtext(element=response.object, xpath='UploadId',
+                             namespace=self.namespace)
+        if upload_id is None:
+            # tencent cos xml has no xmlns tag
+            return findtext(element=response.object, xpath='UploadId')
+        else:
+            return upload_id

     def _upload_multipart_chunks(self, container, object_name, upload_id,
                                  stream, calculate_hash=True):

[rzvoncek]

Hi @dispensable. If this is still relevant, can you please try with a recent medusa (0.19.1)? It uses different libraries to talk to the storage backends.

But in general, this looks like a very low-level thing where a S3-compatible vendor is not exactly compatible. It might be necessary to write a brand new storage implementation for this, but it's hard to tell.

[dispensable]

@rzvoncek hi, sorry for the delay. I have tried new s3 lib. it works pretty well except the tencent cloud need to force s3={"addressing_style": "virtual"} to the boto3 Config. I just submitted a pr to allow user config this opt.

thanks again