overextended / ox_core

Player and vehicle management and persistence for FiveM.
https://overextended.dev/ox_core
GNU Lesser General Public License v3.0
142 stars 109 forks source link

Incorrect text encoding #176

Closed ESK0 closed 2 days ago

ESK0 commented 6 days ago

Hi,

our grade roles contains special characters and ox_core selects them incorrectly from DB. I tried to play with Connection options but without any luck:

Used characters as a group grade label: á, č, ď, é, ě, í, ň, ó, ř, š, ť, ú, ů, ý, ž

{8D9D14F7-C467-438C-99A3-B8786323CCFE} Output: "�, č, ď, �, ě, �, ň, �, ř, š, ť, �, ů, �, ž"

Our database and table is correctly configured to use utf8mb4_unicode {611E2A36-D8D9-415A-9F00-951173A4D976}

When querying label manually everything is selected correctly {0774E85C-FBFD-464A-A684-0F63CC5B6F3C}

ox_core version: 0.31.1 PS: Back in the times of Lua ox_core we didn't have any issues.

PS2: Apparently its only issue with group grades {FB00F0A0-E026-429B-B588-FB9879E87589}

thelindat commented 6 days ago

This seems to be a MariaDB bug that was fixed at some point - couldn't find when exactly, but after 11.1.

Since I select grades as a JSON value, the query is more like:

SELECT JSON_ARRAYAGG(ox_group_grades.label) AS grades FROM ox_group_grades WHERE `group` = 'czech'

image

MariaDB 11.4 is the current LTS and the problem is not present there.


Here's the relevant issue.

https://jira.mariadb.org/browse/MDEV-24784?jql=project%20%3D%20MDEV%20AND%20component%20%3D%20JSON%20AND%20text%20~%20%22JSON_ARRAYAGG%22

ESK0 commented 5 days ago

I was running MariaDB 11.1 and I can confirm updating to 11.4+ solved the issue.

ESK0 commented 2 days ago

Hey there,

after few a days I noticed that the fix with updating the mariadb server only fixed the issue while querying with a external software. Initially I only checked this with a Datagrip {FDD59057-9656-4040-9608-464B6D5A6052}

Unfortunately the server still got the text with incorrect encoding. {D3C590BE-A048-4B99-A697-358A4016BFFB} {3C83BCF3-1B6D-47A5-8DFE-8177DD54C3D3}

Printing 'raw' grades before JSON parse {53F7AFAC-6EA2-402A-9B2E-120C345F6A27}

Running latest ox_core version

PS: I've tried to query the same in Lua with using oxmysql and it works just fine: {E0CAE310-99AA-4995-8AF4-92CA94A98061}

thelindat commented 2 days ago

{E16AC7F9-4CE8-430A-BF47-1D57320ED6FD} {D6DA2D92-DFD5-46B6-8BA1-FC9502C55B09}

ESK0 commented 2 days ago

{E16AC7F9-4CE8-430A-BF47-1D57320ED6FD} {D6DA2D92-DFD5-46B6-8BA1-FC9502C55B09}

I take it as a L... I uploaded the the new build to a different server folder

ESK0 commented 2 days ago

Even after updating the core correctly (using the correct server directory -_-) The issue with encoding is still there... No issues in Datagrip but ox_core is fucked.. Are you using any options in a connection string? I tried setting charset: 'utf8mb4' but no luck there..

@antond15 Could you please try this at your machine tomorrow? I may look dumb but this is weird AF as issue only exists with ox_core using mariadb but not with oxmysql using mysql2..

thelindat commented 2 days ago

Are you using any options in a connection string

Nothing outside of user:password/database.

ESK0 commented 2 days ago

Are you using any options in a connection string

Nothing outside of user:password/database.

Today afternoon I tried a fresh db with a ox_core recipe and it indeed works as expected...