yiisoft / yii2

Yii 2: The Fast, Secure and Professional PHP Framework
http://www.yiiframework.com
BSD 3-Clause "New" or "Revised" License
14.24k stars 6.91k forks source link

MySQL's JSON column is serialized in a wrong way #18390

Open twisted1919 opened 3 years ago

twisted1919 commented 3 years ago

What steps will reproduce the problem?

Install MariaDB 10.3.17, and create a json column, then save a PaymentIntent (https://github.com/stripe/stripe-php/blob/master/lib/PaymentIntent.php ) object into it.

What is the expected result?

{
    "id": "pi_1Hn54bCadkDflulLUsGqg40B",
    "amount": 1000,
    "object": "payment_intent",
    ....
}

What do you get instead?

Stripe\PaymentIntent JSON: {
    "id": "pi_1HoBQoCadkDflulLbveqgcPh",
    "object": "payment_intent",
    "amount": 3000,
    ....
}

Additional info

On our local environment, using MySQL Server version: 8.0.19, the behavior is correct, we get the right value saved in the database column.
What is more interesting, is that if we try to save an actual array, i.e by explicitly calling PaymentIntent::toArray then Yii errors out with:

{name: "PHP Notice", message: "Array to string conversion", code: 8, type: "yii\base\ErrorException",…}
code: 8
file: "/home/dev/httpdocs/web/vendor/yiisoft/yii2/db/ColumnSchema.php"
line: 157
message: "Array to string conversion"
name: "PHP Notice"
stack-trace: [,…]
0: "#0 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/ColumnSchema.php(157): yii\base\ErrorHandler->handleError()"
1: "#1 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/mysql/ColumnSchema.php(49): yii\db\ColumnSchema->typecast()"
2: "#2 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/QueryBuilder.php(380): yii\db\mysql\ColumnSchema->dbTypecast()"
3: "#3 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/mysql/QueryBuilder.php(258): yii\db\QueryBuilder->prepareInsertValues()"
4: "#4 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/QueryBuilder.php(350): yii\db\mysql\QueryBuilder->prepareInsertValues()"
5: "#5 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/Command.php(478): yii\db\QueryBuilder->insert()"
6: "#6 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/Schema.php(432): yii\db\Command->insert()"
7: "#7 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/ActiveRecord.php(600): yii\db\Schema->insert()"
8: "#8 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/ActiveRecord.php(566): yii\db\ActiveRecord->insertInternal()"
9: "#9 /home/dev/httpdocs/web/vendor/yiisoft/yii2/db/BaseActiveRecord.php(678): yii\db\ActiveRecord->insert()"
10: "#10 /home/dev/httpdocs/web/app/api/modules/v1/modules/event/controllers/OrdersController.php(366): yii\db\BaseActiveRecord->save()"
11: "#11 [internal function]: api\modules\v1\modules\event\controllers\OrdersController->actionCharge()"
12: "#12 /home/dev/httpdocs/web/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()"
13: "#13 /home/dev/httpdocs/web/vendor/yiisoft/yii2/base/Controller.php(180): yii\base\InlineAction->runWithParams()"
14: "#14 /home/dev/httpdocs/web/vendor/yiisoft/yii2/base/Module.php(528): yii\base\Controller->runAction()"
15: "#15 /home/dev/httpdocs/web/vendor/yiisoft/yii2/web/Application.php(104): yii\base\Module->runAction()"
16: "#16 /home/dev/httpdocs/web/vendor/yiisoft/yii2/base/Application.php(392): yii\web\Application->handleRequest()"
17: "#17 /home/dev/httpdocs/web/app/init.php(188): yii\base\Application->run()"
18: "#18 /home/dev/httpdocs/web/public_html/api/index.php(15): require('/home/dev/httpd...')"
19: "#19 {main}"
type: "yii\base\ErrorException"

The documentation at https://www.yiiframework.com/doc/guide/2.0/en/db-active-record is not really clear on how exactly we should save json data into a json column, shouldn't it accept an array/object/etc and then just json_encode it when it saves it? Because if that's the case, it isn't working correctly.
I also tried passing a json encoded string, but this resulted into escaped json saved into the database, which at this point i don't know if it's caused by Yii or by the database engine.

Q A
Yii version 2.0.39.2
PHP version 7.4
Operating system CentOS 7
twisted1919 commented 3 years ago

Please hold with this, i think i know what is going on.

twisted1919 commented 3 years ago

Okay, so it's the database engine...

create table testing_json ( the_content JSON );
show create table testing_json 
# result
CREATE TABLE `testing_json` (
 `the_content` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1

So MariaDB silently fails to create the JSON column and creates a longtext instead.
but wait, there is more. If you read the docs at https://mariadb.com/kb/en/json-data-type/ they say:

JSON is an alias for LONGTEXT introduced for compatibility reasons with MySQL's JSON data type. MariaDB implements this as a LONGTEXT rather, as the JSON data type contradicts the SQL standard, and MariaDB's benchmarks indicate that performance is at least equivalent.

So while they implement all the functionality for json, their data type is actually a longtext.
This is why Yii cannot detect the right the column type, after all, for Yii this is a longtext column which only accepts text. This is also the reason why when I pass an array to the property it errors out with the array to string conversion error.

What are our options here?
Could we wrap our values into \yii\db\JsonExpression maybe? Or what options do we have, except changing the database engine from mariadb to mysql?

twisted1919 commented 3 years ago

Additionally, please beware that the casting done by yii\db\mysql\JsonExpressionBuilder::build will fail for mariadb as the JSON data type does not exists.

samdark commented 3 years ago

https://www.yiiframework.com/extension/sam-it/yii2-mariadb?

twisted1919 commented 3 years ago

@samdark - Thanks, i didn't know about that link.
Since I don't actually need to search the json right now, but just store it, I will use the longtext option for now, so from my point of view, you can close this issue, but I think I raised a good problem that should be covered by the framework and documented.

samdark commented 3 years ago

I think we can explicitly link to yii2-mariadb in DB drivers list.

st0at commented 3 years ago

@twisted1919 for now for such problem I used such solution:

class JsonModel extends ActiveRecord
{
    public function beforeSave($insert)
    {
        $this->json_field = json_encode($this->json_field);
        return parent::beforeSave($insert);
    }

    public function afterFind()
    {
        $this->json_field = json_decode($this->json_field);
        parent::afterFind();
    }
}