UnicodeDecodeError: 'utf-8' codec can't decode byte ...

[Thempra]

Long story short

When request a call to some servers, a coding error occurs instead download the image / content

Actual behaviour

Traceback (most recent call last):
  File "aiohttpPoC.py", line 16, in <module>
    loop.run_until_complete(main())
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "aiohttpPoC.py", line 13, in main
    html = await fetch(session, imageurl)
  File "aiohttpPoC.py", line 8, in fetch
    return await response.text()
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 921, in text
    return self._body.decode(encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 6: invalid continuation byte

Steps to reproduce

import aiohttp
import asyncio
import datetime
import pytz

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    imageurl='https://www.aemet.es/imagenes_d/eltiempo/observacion/radar/'+datetime.datetime.now(pytz.timezone('Atlantic/Cape_Verde')).strftime("%Y%m%d%H%M")[:-1]+'0_r8va.gif'
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, imageurl)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Your environment

Python 3.7.0 aiohttp 3.4.4

[asvetlov]

GitMate.io thinks possibly related issues are https://github.com/aio-libs/aiohttp/issues/1731 (UnicodeEncodeError: 'utf-8' codec can't encode character '\udca9'), https://github.com/aio-libs/aiohttp/issues/207 (UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 60: surrogates not allowed), https://github.com/aio-libs/aiohttp/issues/1750 (Encoding is always UTF-8 in POST data), https://github.com/aio-libs/aiohttp/issues/18 (Auto-decoding doesn't recognize content-type: application/json; charset=utf-8), and https://github.com/aio-libs/aiohttp/issues/1802 (response.text() crash if body contain non UTF-8 symbol).

[asvetlov]

You are trying to fetch GIF as UTF-8 encoded text. Obviously, this is an error. What behavior do you expect?

[Thempra]

rewrite again and now it's ok, thx !!

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.content.read()

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. If you feel like there's important points made in this discussion, please include those exceprts into that new issue.