Error message of "Malformed UTF-8" hiding real error messages

[AntonioCS]

API Platform version(s) affected: v2.6.5

Description
I'm getting this error for most problems on api calls:

For this request data:

{
  "order": "api/orders/696e4882-88bf-4b91-984c-ec41f7498958",
  "token": "12345",
  "cardNumber": "xyz",
  "cardExpiryDate": "1125",
  "currency": "CHF",
  "cardSchemeName": "Visa",
  "dataProcessed": "2021-08-25T10:14:32.534Z",
  "rawResponse": "data",
  "createdAt": "2021-08-25T10:14:32.534Z",
  "uuid": "3fa85f64-5717-4562-b3fc-2c963f66af79"
}

The issue here is dataProcessed was renamed to the correct naming of dateProcessed (note the e), but because of, (I'm assuming) the usage of a uuid, there were encoding issues on the exception and that naming issue was hidden, and I could only see it in the logs:

[2021-08-27T20:34:54.262360+00:00] request.CRITICAL: Uncaught PHP Exception Doctrine\DBAL\Exception\NotNullConstraintViolationException: "An exception occurred while executing 'INSERT INTO payment (token, card_number, card_expiry_date, currency, card_scheme_name, date_processed, raw_response, created_at, updated_at, uuid, order_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params ["12345", "xyz", "1125", "CHF", "Visa", null, "data", "2021-08-27 20:34:54", "2021-08-27 20:34:54", "\x3f\xa8\x5f\x64\x57\x17\x45\x62\xb3\xfc\x2c\x96\x3f\x66\xaf\x79", 1]:  SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'date_processed' cannot be null"....

This is causing HUGE issues as every little error will give me the Malformed UTF-8... error message and I have to go into the server and tail the logs o see the actual error.

HELP!

How to reproduce
Have something throw an error where you use uuids

Possible Solution

Additional Context

[Chris53897]

Related https://github.com/api-platform/api-platform/issues/1900#issuecomment-883489939

[Chris53897]

There are 2 Problems here. Info: I am reading you Slack-Messages.

a) The Malformed UTF-8 Message is hiding the real error b) Real Error: Integrity constraint violation: 1048 Column 'date_processed' cannot be null"... I assume your Entity is missing this code = null; /**

• @ORM\Column(type="datetime", nullable=true) */ public ?DateTime $dateProcessed = null;

And there is a misspelling between dataProcessed and dateProcessed.

[AntonioCS]

@Chris53897 the real error was simple to fix the big issue was that it was hidden by the Malformed... error message. And this was an example, I've gotten other errors that I had to go to the logs to see what the actual error was because I always got the Malformed... error message as I'm using UUIDs everywhere

[Chris53897]

That Malformed... is nagging me too. Hopefully we will see an PR to fix this permanent

[AntonioCS]

Well it seems @mkalisz77 provided a simple solution. Is there no way to just pass a different encoder? Someone in symfony slack said there might be way (but failed to provide any details)

[dunglas]

I think that this should be fixed in Doctrine directly. The exception message should not contain raw binary data (the encoded UUID). The binary data should at least be base64encoded or something like this.

[AntonioCS]

@dunglas A solution is offered here: https://github.com/api-platform/api-platform/issues/1900 and it's just he addition of one more flag: JSON_INVALID_UTF8_IGNORE

[dunglas]

Would you mind to open a PR?

[AntonioCS]

@dunglas I hope I did this right: https://github.com/api-platform/core/pull/4416 Don't do much PRs into open source stuff so using the PR system of github to do one cross project is quite new to me (might have done it before but long time ago).

[dunglas]

From the logs, we can see that the binary payload is encoded as an ASCII string: "\x3f\xa8\x5f\x64\x57\x17\x45\x62\xb3\xfc\x2c\x96\x3f\x66\xaf\x79". This is lost somewhere, or done too late. I think that it's the root of the issue. The JSON serializer will be able to serialize this form (but not the raw binary UUID).

[AntonioCS]

To fix this I created a patch file:

--- vendor/api-platform/core/src/Serializer/JsonEncoder.php 2021-09-06 10:49:11.666009789 +0000
+++ patch/ApiPlatform/JsonEncoder.php   2021-09-06 13:49:21.011915801 +0000
@@ -40,7 +40,8 @@
         }

         // Encode <, >, ', &, and " characters in the JSON, making it also safe to be embedded into HTML.
-        $jsonEncodeOptions = \JSON_HEX_TAG | \JSON_HEX_APOS | \JSON_HEX_AMP | \JSON_HEX_QUOT | \JSON_UNESCAPED_UNICODE;
+        $jsonEncodeOptions = \JSON_HEX_TAG | \JSON_HEX_APOS | \JSON_HEX_AMP | \JSON_HEX_QUOT | \JSON_UNESCAPED_UNICODE | \JSON_INVALID_UTF8_IGNORE;
+        // NOTE!!: Added \JSON_INVALID_UTF8_IGNORE above to fix issue with json and uuids in doctrine exceptions
         if (interface_exists(AdvancedNameConverterInterface::class)) {
             $jsonEncode = new JsonEncode(['json_encode_options' => $jsonEncodeOptions]);
             $jsonDecode = new JsonDecode(['json_decode_associative' => true]);

and I just have this in my composer.json:

    "scripts": {
        "post-install-cmd": [
            "@auto-scripts",
            "patch /var/www/<project>/vendor/api-platform/core/src/Serializer/JsonEncoder.php /var/www/<project>/patches/ApiPlatform/JsonEncoder.patch"
        ],

I had to provide the full path due to directory changes and also this: https://unix.stackexchange.com/questions/167216/how-can-i-apply-a-p0-patch-from-any-working-directory

I'm sure this will be fixed eventually in API Plataform or in Doctrine, but I cannot wait any longer, this issue is hiding other issues and so users cannot report errors, I'm the one that has to go in the logs and find the real issue (which is painful). Thank you for you feedback @dunglas and I wish I could have fixed this in a better way.