How to post multipartWriter form using Streaming?

[fangchaooo]

Long story short

I post a from with file datas to S3.

Expected behaviour

I want using Streaming to read file

my code

multipartWriter = aiohttp.MultipartWriter('mixed') 
  multipartWriter.append(res['data']['uploadInfo']['AWSAccessKeyId']).set_content_disposition(
      'form-data', name='AWSAccessKeyId')
  multipartWriter.append(res['data']['uploadInfo']['policy']).set_content_disposition(
      'form-data', name='policy')
  multipartWriter.append(res['data']['uploadInfo']['signature']).set_content_disposition(
      'form-data', name='signature')
  multipartWriter.append(res['data']['uploadInfo']['key']).set_content_disposition(
      'form-data', name='key')
  multipartWriter.append(open(save_path, 'rb'),
                         {"Content-Type": "multipart/form-data"}).set_content_disposition(
      'form-data',
      name='file',
      filename=url_filename)
 async with session.post(res['data']['uploadInfo']['url'], data=multipartWriter,
                                            headers={'Content-Type': 'multipart/form-data; boundary=' + multipartWriter.boundary}) as res:
  if res.status == 200:
      print("success")

[derlih]

Hi @fangchaooo , I've tried to reproduce your issue with:

async def test_multipart_writer_file_open(aiohttp_client: Any) -> None:
    async def handler(request):
        return web.Response()

    app = web.Application()
    app.add_routes([web.post("/", handler)])

    client = await aiohttp_client(app)
    with pathlib.Path(__file__).open("rb") as fobj:
        multipartWriter = aiohttp.MultipartWriter("mixed")
        multipartWriter.append(
            fobj, {"Content-Type": "multipart/form-data"}
        ).set_content_disposition("form-data", name="file", filename="file.py")

        async with await client.post(
            "/",
            data=multipartWriter,
            headers={
                "Content-Type": "multipart/form-data; boundary="
                + multipartWriter.boundary
            },
        ) as resp:
            assert resp.status == 200

But this test passes (at least on master branch)

[fangchaooo]

Hi @fangchaooo , I've tried to reproduce your issue with:

async def test_multipart_writer_file_open(aiohttp_client: Any) -> None:
    async def handler(request):
        return web.Response()

    app = web.Application()
    app.add_routes([web.post("/", handler)])

    client = await aiohttp_client(app)
    with pathlib.Path(__file__).open("rb") as fobj:
        multipartWriter = aiohttp.MultipartWriter("mixed")
        multipartWriter.append(
            fobj, {"Content-Type": "multipart/form-data"}
        ).set_content_disposition("form-data", name="file", filename="file.py")

        async with await client.post(
            "/",
            data=multipartWriter,
            headers={
                "Content-Type": "multipart/form-data; boundary="
                + multipartWriter.boundary
            },
        ) as resp:
            assert resp.status == 200

But this test passes (at least on master branch)

sry， maybe my words is ambiguously phrased. I mean I want using Streaming reader to post form,. As my code, I am direct to open, but I want to open below:

with open(filename, 'wb') as fd:
    while True:
        chunk = await resp.content.read(chunk_size)
        if not chunk:
            break
        fd.write(chunk)

[Dreamsorcerer]

I'm still not clear what the request is. I'd assume that MultiPartWriter will stream the data as the client sends it. Can you provide a complete test or something demonstrating what is wrong?