Testing image upload to mocked S3

[Andrew-Chen-Wang]

Reporting Bugs

I'm writing some test case for working on presigned post image upload. I'm following the code from here:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html#generating-a-presigned-url-to-upload-a-file

With some additional parameters that shouldn't matter. When you create the presigned post, you'll get a dictionary. Following their requests test setup, I upload an image but get this traceback:

    def _bucket_response(self, request, full_url, headers):
        querystring = self._get_querystring(full_url)
        method = request.method
        region_name = parse_region_from_url(full_url)

        bucket_name = self.parse_bucket_name_from_url(request, full_url)
        if not bucket_name:
            # If no bucket specified, list all buckets
            return self.all_buckets()

        self.data["BucketName"] = bucket_name

        if hasattr(request, "body"):
            # Boto
            body = request.body
        else:
            # Flask server
            body = request.data
        if body is None:
            body = b""
        if isinstance(body, six.binary_type):
>           body = body.decode("utf-8")
E           UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 1192: invalid start byte

/usr/local/lib/python3.8/site-packages/moto/s3/responses.py:267: UnicodeDecodeError

Should I be running the mock server instead during my pytests?

[bblommers]

Hi @Andrew-Chen-Wang, thanks for reporting this. You'll likely get the same response when using server mode - as far as I know, we always assume a raw-text upload. Marking this as a bug.

[Andrew-Chen-Wang]

I see. Is it enough to just decode("base64")? Or because it was a form upload, with other texts mixed in, there has to be some more set up involved?

[bblommers]

Hi @Andrew-Chen-Wang, sorry for the delay. Yes, base64 encoding might be enough.

Can you share your unit test, by any chance? I'm unable to reproduce this.

FYI, I do get a similar error (different stack trace) when using this code:

with open("file.jpg") as f:

But that can be fixed by passing in the mode-argument as rb:

with open("file.jpg", "rb") as f:

[Andrew-Chen-Wang]

Whoops accidentally closed.

Edit: actually one thing that could possibly be making this happen is that I'm using Factory Boy's factory.django.ImageField, finding the image's path, and then uploading that file. To reproduce, it's a bit of Django if you don't mind, so I put it at the bottom. I'll also try out the rb mode and see if that helps!

Second Edit: yea even with rb mode, it's still giving the same traceback.

My presigned post was basically the ones shown in Boto3, but as requested/copied from docs:

import logging
import boto3
from botocore.exceptions import ClientError

def create_presigned_post(bucket_name, object_name,
                          fields=None, conditions=None, expiration=3600):
    """Generate a presigned URL S3 POST request to upload a file

    :param bucket_name: string
    :param object_name: string
    :param fields: Dictionary of prefilled form fields
    :param conditions: List of conditions to include in the policy
    :param expiration: Time in seconds for the presigned URL to remain valid
    :return: Dictionary with the following keys:
        url: URL to post to
        fields: Dictionary of form fields and values to submit with the POST
    :return: None if error.
    """

    # Generate a presigned S3 POST URL
    s3_client = boto3.client('s3')
    try:
        response = s3_client.generate_presigned_post(bucket_name,
                                                     object_name,
                                                     Fields=fields,
                                                     Conditions=conditions,
                                                     ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL and required fields
    return response

import requests    # To install: pip install requests

# Generate a presigned S3 POST URL
object_name = 'OBJECT_NAME'
response = create_presigned_post('BUCKET_NAME', object_name)
if response is None:
    exit(1)

# Demonstrate how another Python program can use the presigned URL to upload a file
with open(object_name, 'rb') as f:
    files = {'file': (object_name, f)}
    http_response = requests.post(response['url'], data=response['fields'], files=files)
# If successful, returns HTTP status code 204
logging.info(f'File upload HTTP status code: {http_response.status_code}')

Edit: this is some sample Django code to reproduce the image uploading I was doing:

# models.py
from django.db import models

class Blah(models.Model):
    image = models.ImageField()

from factory import DjangoModelFactory
from factory.django import ImageField

class BlahFactory(DjangoModelFactory):
    image = ImageField()

    class Meta:
        model = Blah

# test case using pytest
@mock_s3
def test_image_upload():
    blah = BlahFactory.create()
    # Follow the steps from above to create presigned post.
    # You can access the image by doing blah.image
    # You can access its path using blah.image.path

[bblommers]

I can't reproduce this I'm afraid. I've set up a sample Django code with just the one app/view. This is the view-code that I use, which looks example the same as your example:

def index(request):
    from blah.models import BlahFactory

    blah = BlahFactory.create()
    image_upload(blah.image.path)
    return HttpResponse("Hello, world:  " + blah.image.path)

@mock_s3
def image_upload(path):
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket="mybucket")

    object_name = "red.jpg"
    response = s3.generate_presigned_post(
        "mybucket",
        object_name,
        Fields=None,
        Conditions=None,
        ExpiresIn=3600
    )

    with open(path, "rb") as f:
        files = {'file': (object_name, f)}
        http_response = requests.post(response['url'], data=response['fields'], files=files)
        print(http_response)

I'm assuming you're on the latest version of moto (1.3.16)?

[bblommers]

Going to close this due to a lack of response. Feel free to reopen if this issue still persists