Obeyed commented 1 month ago

Improve documentation

Link

https://supabase.com/docs/guides/storage/uploads/s3-uploads

Describe the problem

It is not understandable how to prepare the local / self-hosted environment for parallel S3 uploads.

Describe the improvement

Are there any guides available for using the S3 Protocol when self-hosting?

Additional context

Originally posted in https://github.com/supabase/storage/issues/495#issuecomment-2225945817 and later in https://github.com/supabase/storage/discussions/528

When I follow the steps written in the referenced issue, I get the SignatureDoesNotMatch error:

{
    "raw": "{\"metadata\":{},\"code\":\"SignatureDoesNotMatch\",\"httpStatusCode\":403,\"userStatusCode\":400}",
    "name": "Error",
    "message": "The request signature we calculated does not match the signature you provided. Check your key and signing method.",
    "stack": "Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n    at Object.SignatureDoesNotMatch (/app/dist/internal/errors/codes.js:140:39)\n    at Object.<anonymous> (/app/dist/http/plugins/signature-v4.js:72:34)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  }

Standard and Resumable uploads work.

At one point I was getting the following error:

Missing S3 Protocol Access Key ID or Secret Key Environment variables

But that seems to have been my mistake with restarting the containers properly, because I'm not seeing this anymore. Now I get the SignatureDoesNotMatch error.

Any pointers on what I could be doing wrong?

Is there anything more I can share that could help?

Storage config

These are the environment variables for the storage container:

STORAGE_BACKEND: s3

# Removing this seems to have no effect
# S3_PROTOCOL_PREFIX: /storage/v1
S3_PROTOCOL_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
S3_PROTOCOL_ACCESS_KEY_SECRET: ${AWS_SECRET_ACCESS_KEY}

AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}

AWS_DEFAULT_REGION: ${S3_REGION}
REGION: ${S3_REGION}

GLOBAL_S3_BUCKET: ${S3_GLOBAL_S3_BUCKET}

TENANT_ID: ${TENANT_ENVIRONMENT}
IS_MULTITENANT: "false"

TUS_URL_PATH: /upload/resumable
TUS_URL_EXPIRY_MS: 86400000
UPLOAD_SIGNED_URL_EXPIRATION_TIME: 86400000

Kong config

My kong config for the storage-v1 path is as follows:

Notice the /storage/v1 in "Forwarded: host=$(headers.host)/storage/v1;proto=http".

## Storage routes: the storage server manages its own auth
- name: storage-v1
  _comment: "Storage: /storage/v1/* -> http://storage:5000/*"
  connect_timeout: 60
  write_timeout: 3600
  read_timeout: 3600
  url: http://storage:5000/
  routes:
    - name: storage-v1-all
      strip_path: true
      paths:
        - /storage/v1/
      request_buffering: false
      response_buffering: false
  plugins:
    - name: cors
    - name: request-transformer
      config:
        add:
          headers:
            - "Forwarded: host=$(headers.host)/storage/v1;proto=http"

Python Boto3

Example of how I'm doing the upload from a client application with Python and boto3. Couldn't find anything about using Python in the docs, so lots of guessing and trying.

import os
import boto3
from botocore.config import Config

# kong gateway url
gateway_uri = os.getenv("GATEWAY_URI")

s3_client = boto3.client(
    "s3",
    region_name="eu-west-1",
    # assuming this requires the `/storage/v1' otherwise kong can't route it
    endpoint_url=f"{gateway_uri}/storage/v1/s3",
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    # assuming this is how I'd set forcePathStyle as described in the docs
    config=Config(s3={"addressing_style": "path"}),
)

# then do an upload
s3_client.upload_file(
    Filename=file_path,
    Bucket=bucket_name,
    Key=remote_file_path,
)

fenos commented 1 month ago

Hello @Obeyed good timing, i just pushed a new version of Storage that should simplify the S3 protocol on self-host. Could you try using the new image: v1.10.1, without any other config changes the protocol should work.

Previously other setup was necessary. Let me know if it works for you

EDIT:

Additionally, you shouldn't need the additional forwarded header in the kong config