Wrong encoding of special characters? (Only in Python module)

[miggland]

I'm not sure this issue is caused by the Python binding - but it's the only place I have observed it.

This was not the case earlier, but I can not determine what caused the change (*). It may have been an update of InvenTree.

When returning a Company address, any special characters are mumbled, eg: KÃ¥ge expecting Kåge Höxter expecting Höxter

The problem does not appear if any of the following fields includes special characters (ö, ä, å, &):

name
notes
contact

I have not found any other field which this problem affects, in other fields the letters are shown as expected. Also, this problem does not appear when the company name includes special characters itself.

Looking at the api results in the browser also gives the expected return. If I get the same results with curl -X GET -H ... I see what I expect.

(*) Downgrading the python binding to 0.6.1 or 0.7.0 made no difference

Version information:

# Version Information:
InvenTree-Version: 0.7.0
Django Version: 3.2.13
Commit Hash: 044315a
Commit Date: 2022-05-24
Database: postgresql
Debug-Mode: False
Deployed using Docker: True

pip3 list
inventree               0.7.1

[miggland]

This problem also affects StockLocations - any special characters in the description are displayed in the wrong encoding, unless the title also contains special characters.

[SchrodingersGat]

@miggland can you identify any other fields which exhibit this problem, or just:

• company.address
• stocklocation.description

[miggland]

I have not searched systematically, but have not noticed the problem elsewhere yet.

[SchrodingersGat]

@miggland I am unable to recreate this problem with a simple test:

from inventree.api import InvenTreeAPI

from inventree.stock import StockLocation

HOST = 'http://localhost:8000'

api = InvenTreeAPI(
    HOST,
    username='admin',
    password='inventree'
)

location = StockLocation(api, pk=1)

print("Description:", location.description)

Description: Höxter

[miggland]

Just now observed the same problem with a purchase order extra line item. I agree it's weird - and it may have to do with how the data was imported, or a database character set? (Ref: https://www.postgresql.org/docs/current/multibyte.html)

[SchrodingersGat]

But you say that the issue only exhibits on the python interface?

• Do you see the error on the same data, but viewed via the web page?
• Do you see the same issue when retrieving data via the REST API?

[miggland]

No, the issue only appears when getting data via the python interface. A direct view of the REST API in the browser, or via curl, shows up properly. Everything looks fine on the webpage as well.

[SchrodingersGat]

Probably not an issue with the database then. Rather the way python is encoding the result returned from the server.

[SchrodingersGat]

Can you show me the output of pip show requests.

I'm running requests 2.27.1 and the encoding returns fine.

[miggland]

Currently:

Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0

After update:

Name: requests
Version: 2.28.0
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0

Result of test after update: problem solved, I can't reproduce this any longer.