gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.65k stars 220 forks source link

Request Forbidden When Connecting to Azure Storage Account #632

Closed bdeluca-igenius closed 1 month ago

bdeluca-igenius commented 1 month ago

Context: I'm trying to create an object in an Azure Storage Account that's configured with a Private Link. Setting S3PROXY_AUTHORIZATION: "none" works, but as recommended I would like to authenticate the incoming requests. Thanks for the help!

S3Proxy Config:

LOG_LEVEL: "TRACE"
JCLOUDS_AZUREBLOB_AUTH: "azureKey"
JCLOUDS_ENDPOINT: "https://<azure_storage_account>.blob.core.windows.net"
JCLOUDS_IDENTITY: "<azure_storage_account>"
JCLOUDS_CREDENTIAL: "<azure_storage_key>"
JCLOUDS_PROVIDER: "azureblob"
S3PROXY_AUTHORIZATION: "aws-v2-or-v4"
S3PROXY_IDENTITY: "local-identity"
S3PROXY_CREDENTIAL: "local-credential"
S3PROXY_ENDPOINT: "http://0.0.0.0:8080"

Client Script:

import boto3
from boto3.s3.transfer import TransferConfig
from botocore.client import Config

def main():
    bucket_config = Config(signature_version="s3v4")
    max_allowed_upload = (1024**3) * 3
    transfer_config = TransferConfig(multipart_threshold=max_allowed_upload)

    base_url = "http://s3proxy:8080"
    access_key_id = "<azure_storage_account>"
    secret_access_key = "<azure_storage_key>"

    source_file_path = "./test.txt"
    bucket = "testcontainer"
    destination_file_path = "test.txt"

    client = boto3.client(
            "s3",
            endpoint_url=base_url,
            aws_access_key_id=access_key_id,
            aws_secret_access_key=secret_access_key,
            config=bucket_config,
            )

    client.upload_file(
            source_file_path,
            bucket,
            destination_file_path,
            Config=transfer_config,
            )

if __name__ == "__main__":
    main()

S3Proxy logs:

[s3proxy] I 05-06 11:46:00.069 main o.g.s.CrossOriginResourceSharing:100 |::] CORS allowed origins: []
[s3proxy] I 05-06 11:46:00.169 main o.g.s.CrossOriginResourceSharing:101 |::] CORS allowed methods: []
[s3proxy] I 05-06 11:46:00.169 main o.g.s.CrossOriginResourceSharing:102 |::] CORS allowed headers: []
[s3proxy] I 05-06 11:46:00.169 main o.g.s.CrossOriginResourceSharing:103 |::] CORS allow credentials: 
[s3proxy] I 05-06 11:46:04.668 main o.g.s.o.e.jetty.server.Server:384 |::] jetty-11.0.20; built: 2024-01-29T21:04:22.394Z; git: 922f8dc188f7011e60d0361de585fd4ac4d63064; jvm 17.0.10+7
[s3proxy] I 05-06 11:46:05.570 main o.g.s.o.e.j.s.AbstractConnector:376 |::] Started ServerConnector@af78c87{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
[s3proxy] I 05-06 11:46:05.768 main o.g.s.o.e.jetty.server.Server:439 |::] Started Server@53a5e217{STARTING}[11.0.20,sto=0] @46700ms
[s3proxy] D 05-06 11:47:21.569 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:301 |::] request: Request(PUT http://s3proxy:8080/testcontainer/test.txt)@16cc0412
[s3proxy] D 05-06 11:47:21.674 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:2971 |::] sendSimpleErrorResponse: 403 InvalidAccessKeyId Forbidden {}
bdeluca-igenius commented 1 month ago

The solution in my case was to replace S3PROXY_IDENTITY: "local-identity" with S3PROXY_IDENTITY: "<azure_storage_account>" and S3PROXY_CREDENTIAL: "local-credential" with S3PROXY_CREDENTIAL: "<storage_account_key>"