Cannot import image to private marketplace on Ceph multisite S3

[roamingengineer311]

Description I cannot add any image from both Ceph or SSH datastores to private marketplace. S3 is ceph-based multisite deployment with 3 zones and SSL issued by CA and applied to ingress. Marketplace configuration in Opennebula:

ACCESS_KEY_ID="ACCESS_KEY"
AWS="NO"
BUCKET="marketplace"
ENDPOINT="https://endpoint-url"
FORCE_PATH_STYLE="YES"
MARKET_MAD="s3"
REGION="default"
SECRET_ACCESS_KEY="SECRET_KEY"
SIGNATURE_VERSION="s3"

When I am trying to add image to marketplace I saw these in logs

Thu Jun 13 15:10:13 2024 [Z0][MKP][E]: Error parsing marketplace information: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 1830844780, columns 9:19. Monitoring information:
opening connection to endpoint-url:443...
opened
starting SSL for endpoint-url:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
<- "HEAD /marketplace/ HTTP/1.1\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.183.1 ua/2.0 api/s3#1.135.0 os/linux md/x86_64 lang/ruby#2.7.0 md/2.7.0 cfg/retry-mode#legacy\r\nDate: Thu, 13 Jun 2024 15:10:13 
..."
-> "HTTP/1.1 200 OK\r\n"
-> "x-rgw-object-count: 206\r\n"
-> "x-rgw-bytes-used: 17616076800\r\n"
-> "x-rgw-quota-user-size: -1\r\n"
-> "x-rgw-quota-user-objects: -1\r\n"
-> "x-rgw-quota-max-buckets: 1000\r\n"
-> "x-rgw-quota-bucket-size: -1\r\n"
-> "x-rgw-quota-bucket-objects: -1\r\n"
-> "x-amz-request-id: tx000002eedb608b4ddab63-00666b0bd5-6ae4-maincube\r\n"
-> "content-length: 0\r\n"
-> "date: Thu, 13 Jun 2024 15:10:13 GMT\r\n"
-> "\r\n"
Conn keep-alive
<- "GET /sysops-marketplace/?encoding-type=url HTTP/1.1\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.183.1 ua/2.0 api/s3#1.135.0 os/linux md/x86_64 lang/ruby#2.7.0 md/2.7.0 cfg/retry-mode#legacy\r\nDate: Thu, 13 Jun 2024 15:10:13 GMT\r\nAuthorization: ..."
-> "HTTP/1.1 200 OK\r\n"
-> "transfer-encoding: chunked\r\n"
-> "x-amz-request-id: tx000009904eb7dbfeff4ec-00666b0bd5-458a-maincube\r\n"
-> "content-type: application/xml\r\n"
-> "date: Thu, 13 Jun 2024 15:10:13 GMT\r\n"
-> "\r\n"
-> "259\r\n"
reading 601 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><EncodingType>url</EncodingType><Name>marketplace</Name><Prefix></Prefix><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>marketapp-147</Key><LastModified>2024-06-13T14:38:51.849Z</LastModified><ETag>&quot;9946dcd3ea4df833d575ac1df432f941-320&quot;</ETag><Size>10737418240</Size><StorageClass>STANDARD</StorageClass><Owner><ID>sysops-marketplace</ID><DisplayName>Sysops Marketplace</DisplayName></Owner><Type>Normal</Type></Contents><Marker></Marker></ListBucketResult>"
read 601 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn keep-alive
USED_MB="10240"
FREE_MB="1038336"
TOTAL_MB="1048576"

While I connect to bucket with fileon S3 browser I see that image exists in bucket

But returning back to Opennebula, image i try to upload:

oneimage list | grep 100
 100 oneadmin oneadmin test_img                                                                 mc_ceph_im     10G OS    No rdy     0

Its state in marketplace

onemarketapp list | grep 148
 148 test2                                                                               0.0    0M  err  img 06/13/24 base_image    0

To Reproduce

• Create marketplace on Ceph RGW
• Try to upload any image to it

Expected behavior App state in marketplace is ready with absence of any errors.

Details

• Affected Component: Private marketplace (MARKETMAD: s3)
• Version: 6.8.0 Additional context When I try to reproduce this on Opennebula with version 6.4.0 with same S3-storage and configuration everything works as expected.

Progress Status

• [ ] Code committed
• [ ] Testing - QA
• [ ] Documentation (Release notes - resolved issues, compatibility, known issues)