Upload callback reporting many more bytes than file size

[compscidr]

• Async AWS SDK for Python version: 8.0.2
• Python version: 3.6
• Operating System: ubuntu 19.10

Description

Describe what you were trying to get done. Tell us what happened, what went wrong, and what you expected to happen.

Uploading smaller files (on the order of Kbs to several Mbs is accurate). Uploading a large file (268Mb) resulting in inaccurate bytes being reported to the callback - it showed the bytes uploaded as 4.43Gb.

What I Did

Did some further digging, and looked at behavior of the boto3 upload in comparison - it looks like every time it calls the callback it sends the amount of bytes sent since the last time it called the callback, not the cumulative total, which it looks like what the aioboto3 upload is doing.

Also, note - the behavior of the download callbacks matches the original boto3, which is to callback with the bytes since last callback.

Here's some sample code (assuming the ~/.aws/credentials file is setup, and self._bucket matches the bucket name you want to push to)

import boto3
import os
import aioboto3

def callback(self, bytes):
    self._total += bytes;
    print("BYTES: " + str(bytes) + " TOTAL: " + str(self._total))

# aioboto3 version
def putfile(self, filename: str):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(self.putfile_async(filename))

async def putfile_async(self, filename: str):
    objectname = os.path.basename(filename)
    async with aioboto3.client("s3") as s3:
        self._total = 0
        await s3.upload_file(filename, self._bucket, objectname, Callback=self.callback)

# boto3 version
def putfile_sync(self, filename: str):
    objectname = os.path.basename(filename)
    s3_client = boto3.client('s3', region_name=self._region)
    self._total = 0
    response = s3_client.upload_file(filename, self._bucket, objectname, Callback=self.callback)

[terricain]

Fixed in v8.0.3 :)