Certain operations not working with S3-compatible backend

[rkoo19]

Describe the bug

Hi, new to using Boto3 and would appreciate any help I can get. I am using AIStore, an S3-compatible object store and using a botocore patch.

The error message suggests that the endpoint URL is not being correctly propagated for certain operations like download_file and get_object. Given that list_objects_v2 works correctly, I am unsure if the configuration is correct but might not be applied consistently.

Thanks in advance.

Expected Behavior

Expected client object to behave same way for list_objects_v2 and download_file in regards to handling S3 endpoint URL.

Current Behavior

download_file fails without being able to connect to endpoint URL.

Reproduction Steps

import boto3
import logging

# Following import intercepts S3 redirections from AIStore and patch them to: Respect HTTP 301, 302 or 307 redirects using the 'Location' header and fail after 3 redirects for a given operation (rather than just one).
from aistore.botocore_patch import botocore

logging.basicConfig(level=logging.DEBUG)

endpoint_url = 'http://localhost:8080/s3'

s3 = boto3.client(
    's3',
    endpoint_url=endpoint_url,  # AIStore endpoint
    aws_access_key_id='', 
    aws_secret_access_key=''
)

# This works
try:
    response = s3.list_objects_v2(Bucket="abc", Prefix="textfile")
    for obj in response.get('Contents', []):
        print(obj['Key'])
except Exception as e:
    logging.error(f"Error listing objects: {e}")

# This doesn't
s3.download_file(Bucket="abc", Key="textfile1", Filename="./textfile1download")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

2.7.21

Environment details (OS name and version, etc.)

Ubuntu 22.04

[tim-finnigan]

Thanks for reaching out. Can you reproduce the issue using just Boto3/Botocore and not aistore? If so, please share your full debug logs by adding boto3.set_stream_logger('') to your script. Otherwise this may be an issue with aistore that would need to be addressed on their end. We cannot guarantee compatibility with third-party software in cases like this.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[rkoo19]

Hi,

I am closing this issue because an outdated dependency caused a KeyError in the redirection logic. Updating the dependency resolved the issue.

Thank you!

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.