JSONReformat output is not correct

[zhaomaosen]

The output of this JSON format with some special characters is incorrect and truncated, but it can be correctly formatted in vscode. This should be a correct JSON：

{"id":{"$oid":"5f51f7fc1a313f6ce3535b77"},"@timestamp" :"2020-09-03T03:16:17.137Z","log" : {"offset" :543391,"file" : {"path" :"C:\Program Files\rempl\Logs\Remediation.011.etl"}},"message" :"CV'#0#2'SetDisableUXWUAccess'#0#1'DoNotConnectToWindowsUpdate'#0#1'NoAutoUpdate'#0#1'AuOptions'#0#1'isRegisteredWithMU'#0#7'isRegisteredWithWU'#0#7'isRegisteredWithWS'#0#7'isRegisteredWithDCAT'#0#7'isRegisteredWithOther'#0#7'2'#0'0'#0'1'#0'8'#0'.'#0'3'#0'C'#0#0#0'�'#2#0#0#0#0#0#0'mnM5lRlFzUSYZhmx.0'#0'K'#0'E'#0'Y'#0''#0'N'#0'O'#0'T'#0''#0'F'#0'O'#0'U'#0'N'#0'D'#0#0#0'K'#0'E'#0'Y'#0''#0'N'#0'O'#0'T'#0''#0'F'#0'O'#0'U'#0'N'#0'D'#0#0#0'K'#0'E'#0'Y'#0''#0'N'#0'O'#0'T'#0''#0'F'#0'O'#0'U'#0'N'#0'D'#0#0#0'K'#0'E'#0'Y'#0''#0'N'#0'O'#0'T'#0'_'#0'F'#0'O'#0'U'#0'N'#0'D'#0#0#0#0#0#0#0#1#0#0#0#1#0#0#0#0#0#0#0#0#0#0#0'eY'#7#$13'�'#1#0#0#0'��'#1#0'��'#1#0'F�'#3'�'#1#0#0#0#$1C'9&tBsKB�C��8m^��7'#0#$B#1#0#0#0#1#0#0#0#0'�'#0#0#4#0#0#0#3#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0#0'@'#0#$C#0#1#0'3'#0'3'#0'Microsoft.Windows.Remediation'#0#$13#0#1#$1A'sPO蠅�G�����'#4'v�'#0#0#0#0#0'�'#3#$B#0#0#0'�'#3'�'#3'","input" : {"type" :"log"},"agent" : {"name" :"XLY-LIURONG","type" :"filebeat","version" :"7.9.0","hostname" :"XLY-LIURONG","ephemeral_id" :"e3a78cca-67a7-4b7f-a7fd-bee382589b49","id" :"f39d3911-acef-44f7-bfff-08c9bd2e2ef8"},"ecs" : {"version" :"1.5.0"},"host" : {"os" : {"platform" :"windows","version" :"10.0","family" :"windows","name" :"Windows 10 Education","kernel" :"10.0.17134.1667 (WinBuild.160101.0800)","build" :"17134.1667"},"id" :"b2789ed2-a7cd-49e9-8a58-4229c1e43034","ip" : ["fe80::2094:8e7e:e734:8fb9","192.168.0.52","fe80::69e5:ad7c:10c1:586a","169.254.88.106","fe80::d5e7:eedf:d448:d6f5","192.168.202.1","fe80::15d5:993:d2d2:54d3","192.168.40.1"],"name" :"XLY-LIURONG","mac" : ["34:97:f6:29:8e:76","0a:00:27:00:00:13","00:50:56:c0:00:01","00:50:56:c0:00:08"],"hostname" :"XLY-LIURONG","architecture" :"x86_64"}}

mORMot master, Delphi 10.2.3, Win 10 LTSC 2019 I'm sorry,I can't reproduce this problem. It seems to only appear in my program. When I copy the JSON into a file and read it from the file, it is properly formatted!

[synopse]

There is not enough information. Which version of mORMot are you using? On which compiler? On which OS? Please create a reproduceable example first.

[zhaomaosen]

Any suggestions? Or can we get some error messages when jsonreformat fails

[synopse]

Please put the original JSON before reformat in a https://gist.github.com/ and I will try to find out what's wrong.

[zhaomaosen]

It is possible that there is a #0 in JSON and the pointer is truncated, but in my independent test program, the pointer is not truncated

gist https://gist.github.com/zhaomaosen/aeeea1ef667e140489170338f83fbcdf

[synopse]

So if I understand correctly, there are #0 in the input JSON? This is not valid JSON for sure. \u0000 may be used to store some #0 byte in the JSON, but it is error prone, and a lot of parsers will have problems with it. mORMot will have troubles will it for sure. I don't see how mORMot may support it. The idea is to use base-64 encoding for such binary content.

If indeed you supplied the output JSON, please create a gist with the input JSON, not the ouput JSON.

[zhaomaosen]

Replace the #0 in the JSON string with ‘\u0000’, This problem has been solved. I don't store binary, but there are escape characters in the string. But I hope mormot can handle it correctly, because it is only an escape character. The string in JSON should be wrapped in double quotation marks, so it should end with a double quotation mark instead of #0, or replace it with \u0000 in the parsing result Is a method or gives an error message when parsing an error instead of directly truncating it. Otherwise, users will be confused. Is my JSON wrong or mormot wrong?

[zhaomaosen]

I found that when the key contains the character $ , and there is no double quotation mark, the format is not correct

https://gist.github.com/zhaomaosen/18a99ed6f032900fe842a36535eaa4c1

[synopse]

This is as expected. Your input text is not JSON at all. Input should be JSON, not JavaScript. Function name is JsonReformat() not JavaScriptToJson(). ;)

We only allow basic "MongoDB extended JSON syntax" as relaxed input. The extended jcJsonIdentifier set is ['_', '0'..'9', 'a'..'z', 'A'..'Z', '.', '[', ']'] so $ or * are clearly not supported. You need to double quote the field name.