[BUG] [SP-API] [Issue] Character encoding mismatch in listings that use UTF-8 characters in title or bullet points

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Feature request
[ ] Documentation issue or request
[ ] Other... Please describe:

SP-API Name and Version or Report/Feed Type

Listing Items v2021-08-01 https://developer-docs.amazon.com/sp-api/docs/listings-items-api-v2021-08-01-reference#get-listings2021-08-01itemsselleridsku

Expected Behavior

The Listing Title, Bullet Points and description should be returned with the correct encoding and match what is live on Amazon.

Current Behavior

The listing returned by the API may return incorrectly encoded binary content depending on the SKU and presumably how the listing translation was generated or uploaded. The reported/detected encoding, since the Content-Type is 'application/json', is implicitly utf-8, but decoding the request sometimes returns weird symbols in the listing text or title as if parts of it were in a different encoding "raw_unicode_escape", which is an internal encoding used in python. This only happens on some of the products that use utf-8 characters in their title or bullet points, but not all, and the encoding may be different between title and bullet points. Different fields "seem" to have different encoding, as the item_name may decode correctly while the bullet points will still have incorrect encoding. Since the request response is decoded all at once and not field by field, this always gives back malformed content for the affected listings. I have tested the returned value for multiple SKUs in the German marketplace, ID A1PA6795UKMFR9 I will cover part of the request since that would include Seller ID and mSKU.

Request: GET https://sellingpartnerapi-eu.amazon.com/listings/2021-08-01/items/{sellerId}/{sellerSku}?issueLocale=en_US&marketplaceIds=A1PA6795UKMFR9&includedData=summaries%2Cattributes%2Cissues%2Coffers%2CfulfillmentAvailability

Cut down response: { "sku": {sellerSku}, "summaries": [ { "marketplaceId": "A1PA6795UKMFR9", "asin": {asin}, "productType": "THERMOMETER", "conditionType": "new_new", "status": [ "DISCOVERABLE" ], "fnSku": {fnsku}, "itemName": ... "Infrarot-Thermometer f\u00fcr S\u00e4uglinge, Kinder und Erwachsene"..., "createdDate": "2020-12-01T21:41:05.514Z", "lastUpdatedDate": "2022-06-30T06:00:02.152Z", ...

Context/Screenshots

Thermometer title on Amazon: Thermometer title on SellerCentral - Edit Listing interface Part of binary response: Extracting specific field after converting the response to json and extracting one of the problematic attributes: Fix by encoding and decoding again: .encode('raw_unicode_escape').decode('utf-8')

But when the title has been decoded correctly, the bullet points may still not be: ..."bullet_point":[{"language_tag":"de_DE","value":"... SchlieÃŸlich mÃ¼ssen Sie wissen, wann Sie sich entspannen und wann Sie besonders aufpassen und vielleicht den Arzt rufen sollten. Das {brandName} Baby-Fieberthermometer wurde mit hÃ¶chster PrÃ¤zision entwickelt, damit Sie und Ihr Baby sich entspannen kÃ¶nnen.","marketplace_id":"A1PA6795UKMFR9"},

Attempting the same "conversion" that worked with the title to the bullet points returns an error:

Different listings for the same seller in the same market with the same "utf-8" characters return the correct listing text and title through the SP API:

Your Environment

I'm using the python-amazon-sp-api client library which uses python's requests module internally to make API calls. Python version 3.11. The returned response's binary content is decoded to text, then read into a dictionary using the json() method (from the simplejson library). The title or bullet point text is not malformed by the conversion to dict using the json method.

amzn / selling-partner-api-models