json import/export/edit problem with utf-8 encoding (or something)

[SplinterGU]

for example, characters \u00f3 are changed by \xf3, this produce invalid text

this must be "Estación ..."

            "TeleporterType": "Spacestation",
            "Name": "Estaci\xF3n menor Taylor-Omnd",
            "CalcWarpOffset": false,
            "IsFeatured": false

correct value:

            "TeleporterType": "Spacestation",
            "Name": "Estaci\u00F3n menor Taylor-Omnd",
            "CalcWarpOffset": false,
            "IsFeatured": false

same issue for others characters... for example,

            "TeleporterType": "Spacestation",
            "Name": "T\xE9rmino de Jonareham",
            "CalcWarpOffset": false,
            "IsFeatured": false
        },

must be

            "TeleporterType": "Spacestation",
            "Name": "T\u00E9rmino de Jonareham",
            "CalcWarpOffset": false,
            "IsFeatured": false
        },

[ihleslie]

May be related. When trying to name a spaceship using unicode, it is not possible to have more than one character. Specifically, anything after the first unicode character is not shown. It looks like the null code (\u000) is being added when saving the JSON file. Subsequent saves add additional null codes. The image below shows the before and after. BTW, the NomNom editor did allow two unicode characters, but caused other problems for me. I like your editor.

[goatfungus]

May be related. When trying to name a spaceship using unicode.....

There is a current bug with the encoder which I am in the process of fixing. Some characters work fine (those that encode to 3-bytes) but some result in a zero byte at the end of the sequence. I don't think this is related to the original issue though, as this is to do with the \uFFFF encoded characters.

\xFF is another thing entirely. It is basically a representation of a non-unicode byte value that is present in the save file, so if you use it be aware that you are not actually representing a character. Some things in the file, such as inventory item codes, are not in unicode but are a binary string, so this is my effort to at least support that.

[goatfungus]

Correction: the bug mentioned above IS causing the \xFF encoding for some characters, but the next update will solve both problems. Thanks for reporting this.

[goatfungus]

This should all be fixed now.

[SplinterGU]

now I have a new error when I edit the RAW JSON

[ihleslie]

@SplinterGU, Just throwing this out. Your unicodes appear to be for UTF-16 encoding. There are at least some UTF-8 equivalents. For example; \u2CA6 is the Greek/Coptic letter cap Tau.

\u03A4 is the UTF-8 equivalent. I am using the unicode list on wikipedia. https://en.wikipedia.org/wiki/List_of_Unicode_characters#

Cheers

On Fri, Apr 28, 2023 at 7:08 AM SplinterGU @.***> wrote:

now I have a new error when I edit the RAW JSON

[image: image] https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png

— Reply to this email directly, view it on GitHub https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527540453, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.***>

[SplinterGU]

@SplinterGU, Just throwing this out. Your unicodes appear to be for UTF-16 encoding. There are at least some UTF-8 equivalents. For example; \u2CA6 is the Greek/Coptic letter cap Tau. \u03A4 is the UTF-8 equivalent. I am using the unicode list on wikipedia. https://en.wikipedia.org/wiki/List_of_Unicode_characters# Cheers … On Fri, Apr 28, 2023 at 7:08 AM SplinterGU @.> wrote: now I have a new error when I edit the RAW JSON [image: image] https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png — Reply to this email directly, view it on GitHub <#805 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.>

then? what is the solution? I only edit the JSON, I don't change nothing

[ihleslie]

I would try UTF-8 codes and see what happens. I never tried changing owner name.

On Fri, Apr 28, 2023, 8:28 AM SplinterGU @.***> wrote:

@SplinterGU https://github.com/SplinterGU, Just throwing this out. Your unicodes appear to be for UTF-16 encoding. There are at least some UTF-8 equivalents. For example; \u2CA6 is the Greek/Coptic letter cap Tau. \u03A4 is the UTF-8 equivalent. I am using the unicode list on wikipedia. https://en.wikipedia.org/wiki/List_of_Unicode_characters# Cheers … <#m5578296619413084558> On Fri, Apr 28, 2023 at 7:08 AM SplinterGU @.> wrote: now I have a new error when I edit the RAW JSON [image: image] https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png — Reply to this email directly, view it on GitHub <#805 (comment) https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527540453>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.>

then? what is the solution?

— Reply to this email directly, view it on GitHub https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527656349, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YTZR3QSQLC2XOD5RZLXDPH2LANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.***>

[SplinterGU]

I would try UTF-8 codes and see what happens. I never tried changing owner name. … On Fri, Apr 28, 2023, 8:28 AM SplinterGU @.> wrote: @SplinterGU https://github.com/SplinterGU, Just throwing this out. Your unicodes appear to be for UTF-16 encoding. There are at least some UTF-8 equivalents. For example; \u2CA6 is the Greek/Coptic letter cap Tau. \u03A4 is the UTF-8 equivalent. I am using the unicode list on wikipedia. https://en.wikipedia.org/wiki/List_of_Unicode_characters# Cheers … <#m5578296619413084558> On Fri, Apr 28, 2023 at 7:08 AM SplinterGU @.> wrote: now I have a new error when I edit the RAW JSON [image: image] https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png — Reply to this email directly, view it on GitHub <#805 (comment) <#805 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.> then? what is the solution? — Reply to this email directly, view it on GitHub <#805 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YTZR3QSQLC2XOD5RZLXDPH2LANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.>

I din't change the owner name... I only edit other text in other section (name of a Station)

this is a new bug from the last change

[ihleslie]

In my save file "USN": is followed by my Steam user name. If you are trying to change the base name, then go a few lines down to "Name":

On Fri, Apr 28, 2023 at 8:58 AM SplinterGU @.***> wrote:

I would try UTF-8 codes and see what happens. I never tried changing owner name. … <#m4779869165612850705> On Fri, Apr 28, 2023, 8:28 AM SplinterGU @.*> wrote: @SplinterGU https://github.com/SplinterGU https://github.com/SplinterGU https://github.com/SplinterGU, Just throwing this out. Your unicodes appear to be for UTF-16 encoding. There are at least some UTF-8 equivalents. For example; \u2CA6 is the Greek/Coptic letter cap Tau. \u03A4 is the UTF-8 equivalent. I am using the unicode list on wikipedia. https://en.wikipedia.org/wiki/List_of_Unicode_characters# https://en.wikipedia.org/wiki/List_of_Unicode_characters# Cheers … <#m5578296619413084558> On Fri, Apr 28, 2023 at 7:08 AM SplinterGU @.> wrote: now I have a new error when I edit the RAW JSON [image: image] https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png https://user-images.githubusercontent.com/95715426/235156026-f5b4fae6-d77b-45d9-86aa-efe5d97e213f.png — Reply to this email directly, view it on GitHub <#805 https://github.com/goatfungus/NMSSaveEditor/issues/805 (comment) <#805 (comment) https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527540453>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YRUO6QVMID7MXC6DNDXDO6OLANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.> then? what is the solution? — Reply to this email directly, view it on GitHub <#805 (comment) https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527656349>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YTZR3QSQLC2XOD5RZLXDPH2LANCNFSM6AAAAAAXNHVIWU https://github.com/notifications/unsubscribe-auth/A7PF7YTZR3QSQLC2XOD5RZLXDPH2LANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.***>

I din't change the owner name... I only edit other text in other section (name of a Station)

— Reply to this email directly, view it on GitHub https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527695985, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YXKOYI5GJFDNE7LQOLXDPLJBANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.***>

[SplinterGU]

sorry, for my bad english...

I only change text in "Name":, but in other item, no in item error... for example I change the text in line 489216 and error was in line 564046... not same item... sorry I don't understand that you mean, or I can't explain me...

[ihleslie]

The line numbers do not mean anything to me because each save file is different. I do not know if I can help, but if you send me screenshots of where the change was made and where the error appeared, that might help. Please expand the file structure as much as possible so I can find the equivalent entries in my save file. See the attached image for what I mean.

On Fri, Apr 28, 2023 at 11:28 AM SplinterGU @.***> wrote:

sorry, for my bad english...

I only change text in "Name":, but in other item, no in item error... for example I change the text in line 489216 and error was in line 564046... not same item... sorry I don't understand that you mean, or I can't explain me...

— Reply to this email directly, view it on GitHub https://github.com/goatfungus/NMSSaveEditor/issues/805#issuecomment-1527874003, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7PF7YU4KBMZB25MEYVFQ2DXDP42JANCNFSM6AAAAAAXNHVIWU . You are receiving this because you commented.Message ID: @.***>

[goatfungus]

Ok, i found out what the problem is and have a fix coming out shortly.

@ihleslie javascript/json string escapes (\uFFFF) are in UTF-16 format according to the specifications. So what you are seeing in the JSON editor is technically correct, so once the next update is pushed it should all work fine.

One thing to note is the reason that I'm even using \uFFFF string escapes for the JSON Editor in the first place is because the text control itself doesn't play well with full unicode. I've had to limit the display to ASCII characters only. However, if you copy blocks of text out of the editor into notepad++ or something equivalent, the copy function should automatically convert it all to UTF-8 and back again. Also, any JSON export/import functions in the editor will only use \uFFFF string escapes for the required characters in the 0x00-0x1f range, all others will be in UTF-8 format.

[goatfungus]

I have updated the JSON Editor to be more resilient to string escapes in version 1.13.8. Hope this works for you now.

[SplinterGU]

I have updated the JSON Editor to be more resilient to string escapes in version 1.13.8. Hope this works for you now.

I'll test it! thanks!

[SplinterGU]

I have updated the JSON Editor to be more resilient to string escapes in version 1.13.8. Hope this works for you now.

I already test it! Now works! I only must edited old invalid values (\xf3, \xe9, etc) and change it for the correct value (\u00fe, \u00e9, etc)... now works fine and the strings was fixed!

thanks! great job!