S3 upload_fileobj aiofiles handling

[frosthamster]

• Async AWS SDK for Python version: 8.2.0
• Aiofiles version: 0.6.0
• Python version: 3.8.6
• Operating System: macos 0.15.3 (19D76) & CentOS Linux 7 (Core)

Description

I get an error when passing an aiofiles obj to the s3 client upload_fileobj because this condition doesn't work as expected https://github.com/terrycain/aioboto3/blob/713494e06dbbfdbd377bd5b2774adcd8870a5bc6/aioboto3/s3/inject.py#L235

A possible solution is replacing asyncio.iscoroutinefunction with inspect.isawaitable to handle any awaitable objects

# noinspection PyUnresolvedReferences
data = Fileobj.read(io_chunksize)
if inspect.isawaitable(data):  # handles if we pass in aiofiles obj
    data = await data
else:
    await asyncio.sleep(0.0)

What I Did

import asyncio
import inspect
import os

import aioboto3
import aiofiles

async def main():
    async with aiofiles.open('test.txt', 'w') as fd:
        await fd.write('hello world\n')

    async with aiofiles.open('test.txt', 'rb') as fd:
        assert not asyncio.iscoroutinefunction(fd.read)  # this doesn't work as expected

        data = fd.read()
        assert inspect.isawaitable(data)

    async with aioboto3.client(
        service_name='s3',
        aws_access_key_id='...',
        aws_secret_access_key='...',
        endpoint_url='...',
    ) as s3:
        async with aiofiles.open('test.txt', 'rb') as fd:
            await s3.upload_fileobj(fd, 'bucket', 'test.txt')

    os.remove('test.txt')

asyncio.run(main())

Traceback

Traceback (most recent call last):
  File "/Users/anisimov/dev/test/main.py", line 31, in <module>
    asyncio.run(main())
  File "/Users/anisimov/.pyenv/versions/3.8.6/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/anisimov/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/anisimov/dev/test/main.py", line 26, in main
    await s3.upload_fileobj(fd, 'izmena', 'test.txt')
  File "/Users/anisimov/dev/test/.venv/lib/python3.8/site-packages/aioboto3/s3/inject.py", line 273, in upload_fileobj
    await file_reader_future
  File "/Users/anisimov/dev/test/.venv/lib/python3.8/site-packages/aioboto3/s3/inject.py", line 254, in file_reader
    multipart_payload += data
TypeError: can't concat generator to bytes

[terricain]

Nice, yeah I see no issue with changing that to isawaitable

[terricain]

Am going to do some more updates, mainly dropping travis for github actions, will do a release once that's working.

[terricain]

Should be solved in v8.2.1

[frosthamster]

Fine, thanks