azist / azos

A to Z Sky Operating System / Microservice Chassis Framework
MIT License
213 stars 29 forks source link

Proposal: add primitive string type hints to JSON ser/deser to roundtrip types correctly #864

Closed itadapter closed 1 year ago

itadapter commented 1 year ago

This is needed to improve performance.

When I write anonymous object, many foundational types are written as string, but when I read it back into JsonDataMap I get back strings, hence, I can not know that originally an atom was written (as an example) or a true short string.

The use of JSON is ubiquitous however it loses type info and intent when we use: new{ a =atom, b=entityId} as both get serialized as strings. If we want to parse json and store as binary later we would not know that a above was Atom - the data is lost today. If we add type hints and a special preservation mode, we can significantly improve the semantical coherence of data across boundaries while still using json as-is.

In other words, we can add system prefixes that will define what plain JSON strings really mean. We can then create Fact archives from JSON content using originally-reported atoms, EntityIds and Gdids, not their string values which are horribbly inefficient (to use). While EntityId can be pattern matched, atoms can not. The reader does not know what was meant to be passed

{
 "a":    1,
 "code": "$atom:sql123" //this will be read as (Atom)"sql123"
}

JsonWritingOptions.EnablePrimitiveStringTypeHints: bool = false (only needed for json datamap) JsonReadingOptions.EnablePrimitiveStringTypeHints: bool = false

Also add JsonReader.GetStringWithTypeHint(string): object or JsonDataMap.InterpretTypeHints(bool recurse)

Only these types are supported on string values:

str - string escape, e.g.  "$str:$aaaaa" is a string "$aaaaa"
bin - binary buffer(base64), not a string
dtm
atm
gd
rgd
eid
latlng
itadapter commented 1 year ago

Must add new unit tests for type hints

itadapter commented 1 year ago
//Plain json:
{
  "Atom": "abc",
  "Eid": "type.Schema@sys::address1",
  "Gdid": "123:0:456789",
  "Rgdid": "456:321:0:98765",
  "Guid": "629867ec-bbd5-40f1-8559-4de34c53a52a",
  "Dt": "1980-01-02T12:30:00Z",
  "Ts": "01:16:07",
  "Bin": "AQECAgMEBQYHCAkADA0OAQUQERL_gWQAAAB75_76xgoAAgQIEw",
  "Str1": "plain string",
  "Str2": "$ZZZ:needs escape"
}

//vs a Hinted json: ------------------------------------------------
{
  "Atom": "$atm:abc",
  "Eid": "$eid:type.Schema@sys::address1",
  "Gdid": "$gdi:123:0:456789",
  "Rgdid": "$rgd:456:321:0:98765",
  "Guid": "$uid:629867ec-bbd5-40f1-8559-4de34c53a52a",
  "Dt": "$dtm:1980-01-02T12:30:00Z",
  "Ts": "$tsp:01:16:07",
  "Bin": "$bin:AQECAgMEBQYHCAkADA0OAQUQERL_gWQAAAB75_76xgoAAgQIEw",
  "Str1": "plain string",
  "Str2": "$str:$ZZZ:needs escape"
}
itadapter commented 1 year ago

Perf tests are the same. Unit tests are passing. Merging into master