`Json` and `MutJson`

[ekeren]

Community Note

Please vote by adding a 👍 reaction to the issue to help us prioritize. If you are interested to work on this issue, please leave a comment.

Feature Spec

Note: this should replace this section of the spec, so please update this section as part of this task.

We introduce two new built-in types Json and MutJson:

let json = Json {
  a: "some string",
  b: 15, 
  c: [ "a", "mb"], 
  j: { 
    a: "sss",
  }
};

let mut_json: MutJson = new MutJson(json);
mut_json.d = "this is allowed"; 
let j : Json = mut_json.to_json();

Use Cases

A couple of question to consider:

• How do we convert Wing's struct from and to Json?
• What is the API to parse string to Json?
• Do we support both json.a and json['a'] and json.get("a")?
• How does the code looks like to get the string value from a path, e.g.:

  let j = { 
  a: "something"
  };
  let some_value: Json = j.get("a"); // get method returns a Json object
  let str_value = some_value.str_value(); // ?? what to do if a is not a string

• Another question I might add is how the compiler should choose to infer whether { "a": 1, "b": 2 } is a Json or Map (or if we should avoid guessing/inference altogether -- so the user is required to specify Json { ... } or Mut { ... }).

Implementation Notes

Originally this type was called Struct

Component

Compiler, SDK

Sub tasks list by @hasanaburayyan

• [ ] Assignment
  • [x] Literals (P1)
  • [x] From native types (P1)
  • [x] To native types from_json() (P1)
    • [ ] unsafe mode (P2)
    • [ ] try_from_json() (P2)
• [x] Json nested properties (P1)
  • [x] get and set
  • [ ] #1680
  • [ ] [] index notation
• [x] Print Json (directly + in string interpolation) (P1)
• [x] #1679
  • [x] Json.clone() && Json.clone_mut() (P1)
  • [x] Json.keys() (P1)
  • [x] Json.values() (P1)
  • [x] Json.delete() (P1)
  • [x] Json.to_str() (P1)
  • [x] Json.parse() (P1)
  • [ ] Json.equals() (P2)
  • [ ] Json.diff() (P2)
  • [ ] Json.patch() (P2)
  • [ ] Json.entries() (P2)
• [ ] Related
  • [x] #1497 (P1)
  • [ ] Structs Schema (P2)

All p2 sub-tasks have been aggregated into #1737

[Chriscbr]

You aren't talking about removing structs (as in struct Person { name: str; ... }) but a different type in the spec called Struct with a capital S, is that right?

[ekeren]

You aren't talking about removing structs (as in struct Person { name: str; ... }) but a different type in the spec called Struct with a capital S, is that right?

Right. It sounds like I need to edit the description to make this clearer, @Chriscbr any suggestion?

[Chriscbr]

I would mention in the description that there is another type with the same, and maybe link to the sections just in case. 👍

Love the proposal btw, I totally see how it fits with our other types like Array/MutArray.

Another question I might add is how the compiler should choose to infer whether { "a": 1, "b": 2 } is a Json or Map<num> (or if we should avoid guessing/inference altogether -- so the user is required to specify Json { ... } or Mut<num> { ... }).

[marciocadev]

let j = { 
  a: "something"
};
let some_value: Json = j.get("a"); // get method returns a Json object
let str_value = some_value.str_value(); // ?? what to do if a is not a string

I don't think a value inside a Json object should be considered a json too, maybe a json component type that can be obtained by a specific call

somethink like

let some_value: JsonComponent =  j.get("a");
print(some_value.type()); // return if it is a string, number, another json, etc.
// to get the value can be using
print(some_value.value()); 
// or
print(some_value);

[marciocadev]

Personally I don't really like the j.get("a") syntax

I find it too verbose

But in a object like

let j = {
  s: {
    o: {
      n: "this is fun"
    } 
  }
};

It's strange get the text with print(j.s.o.n)

[eladb]

Personally I don't really like the j.get("a") syntax

👍 I think we should eventually provide JS-like access to Json objects (j.s.o.n), but we can start without it and iterate.

It's the same array at() (we should allow arr[1] instead of arr.at[1]).

[eladb]

We have a conflict between Json and Map initialization.

I am assuming we want json to be initialized like so:

let json = {
  foo: "foo",
  bar: {
    hey: 12
  }
};

// or with an explicit type
let json = Json { foo: 123 };

// or mutable
let json = { foo: 123 }.to_mut();

Currently, maps are initialized like this:

let m = Map<str>{ "hello": "world" }; // explicit type
let x = { "hello": 12, "bam": 123 };  // inferred type

So the literal { } is currently used for map initialization.

I am assuming we want json to be like json...

So we need to change map. Here are some alternatives:

1. Use a different syntax for maps:

   let m = { "hello" => 2, "world" => 3 };

2. Allow json to be converted to an immutable map (this might be problematic because type checking the value will only happen during preflight).

   let m: Map<str> = { "hello": "world" }.to_map();

3. Any other ideas?

(I am leaning towards 1).

[Chriscbr]

{} isn't just used for Map and Json, it's also for struct types, right?

struct Greeting {
  message: str;
}

let a = Map<str> { "message": "hello" };
let b = Json { "message": "hello" };
let c = Greeting { message: "hello" };

[Chriscbr]

Oops not sure how I closed by accident..

[eladb]

Yes nice catch. We need to somehow distinguish those as well...

Any ideas?

[MarkMcCulloh]

Another idea:

What if Json literals were the only thing with a shorthand instantiation syntax. If we then add implicit (but strong) casting rules only for Json, we can have a pretty flexible but consistent experience when it comes to creating all other types.

struct Thingy {
  a: str;
  b: str;
}

// type: Json
let json_obj = { a: "one", b: "two" };

// type: Json
let json_array = ["one", "two"];

// type: Map<str>
let map1: Map<str> = { a: "one", b: "two" };
let map2: Map<str> = json_obj;
let map3 = new Map<str>({ a: "one", b: "two" });

// type: Array<str>
let array1: Array<str> = ["one", "two"];
let array2: Array<str> = json_array;
let array3 = new Array<str>(json_array);

// type: Thingy
let struct1: Thingy = { a: "one", b: "two" };
let struct2: Thingy = json_obj;
let struct3 = new Thingy(json_obj);

// Unlike other types, Sets actually have to create a copy. 
// This problem is not limited to the suggestion in this post though
// type: Set<str>
//let set1: Set<str> = ["one", "two"];
//let set2: Set<str> = json_array;
//let set3 = new Set<str>(json_array);

[Chriscbr]

I like Mark's idea. The proposal for another syntax for maps could also be fun to try out (though I think => tends to be most associated with functions/closures).

Another option could be to start with a simple rule like "always require a type in front of a braced expression". So you always have to write Map<num> { ... }, Json { }, MyStruct { }, Set { } etc. This way we can just focus on making sure all of the base types feel good / work well.

Later, we can lax the rules and add specific inference rules based on the braced expression's context (like type annotations on a variable declaration, or type parameters of a function) so you can omit the type name in front in some situations.

[MarkMcCulloh]

For inspiration, C# handles this pretty well I think https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/object-and-collection-initializers

[eladb]

@MarkMcCulloh interesting direction, but I kind of feel this breaks the model:

// type: Json
let json_array = ["one", "two"];

This should be an immutable array I think...

I think what we can do is:

1. Use { "foo" => 123 } for maps (this is BTW, how V8 displays map literals).
2. Use { ... } syntax for structs and Json and add explicit support for type checking the structs.

So basically:

struct Boom { a: num; b: str }

let my_map = { "foo" => 12, "bar" => 33 };  // type: Map<num>
let my_json = { hello: 12, world: 88 };     // type: Json
let my_boom = Boom { a: 12, b: "hello" };   // type: Boom

Thoughts?

[MarkMcCulloh]

let json_array = ["one", "two"]; This should be an immutable array I think...

[] is valid JSON so it's actually consistent with the {} literal and us calling it Json. It is atleast immutable though since Json is. That's kinda what's nice about this, if you use the cool syntax then you are always blessed with a Json object.

Use { "foo" => 123 } for maps (this is BTW, how V8 displays map literals).

That's pretty strange looking to me personally. I could see { "foo" = 123 } maybe.

Honestly though if we have a syntax like { hello: 12, world: 88 } for Json why would I ever even want to construct a map? My guess is that I would rather create Jsons all over the place and pass it around expecting it to be coerced to Map as needed.

let my_boom = Boom { a: 12, b: "hello" }; // type: Boom

If this is strictly a struct feature then I'm good with this since internally it's just sugar around Json casting.

[eladb]

[] is valid JSON

Now I see your mental model and the misalignment.

I was thinking Json would only be used for objects, not other types.

If we take the broader definition, then we could argue that "hello" and true are also valid JSON, and then everything is Json...

But it's a good point that if Json only represents objects, then how do we represent JSON arrays and how do we interact with these types?

Need more thinking!

if we have a syntax like { hello: 12, world: 88 } for Json why would I ever even want to construct a map?

Yes that's reasonable. Only that Json is not strongly typed.

I guess we need some more thinking here. We have a few holes in the model...

[eladb]

I guess we can just say that Json must be explicitly stated:

let s = Json "hello";
let a = Json [ "foo", 5 ];
let o = Json { goo: [ 6 ] };
let b = Json true;
let p = Person { name: "Jo" };

Json ser/deser:

let j = Json.parse("[1,2,3]");
let s = Json.to_str(j, pretty: true);

Now let's talk about how structs interact with Json:

let g = Json { name: "queen" };

// check that "g" is valid (schema validation)
Person.is_valid_json(g);

// parse with validation
let p2 = Person.from_json(g);

// parse without validation
let p3 = Person.from_json(g, unsafe: true);

// schema
let schema: JsonSchema = Person.json_schema();

// to json
let pj = p3.to_json();

On the same vain, I guess it makes sense to support these:

let j = download_json();
let a1 = Array<num>.from_json(j);

// without schema validation
let a2 = Array<str>.from_json(j, unsafe: true);
let j2 = a2.to_json();

let n = Num.from_json(j);
let s = Str.from_json(j);

Alternative naming could be from_j/to_j or parse/format.

I would start with the longer names.

[MarkMcCulloh]

I like this approach 👍 Also reminds me of Ballerina's json https://ballerina.io/learn/by-example/json-type/

[ekeren]

@eladb

Any reason for using

let s = Json.to_str(j, pretty: true);

vs

let s = j.to_str(pretty: true);

?

[eladb]

@eladb

Any reason for using

let s = Json.to_str(j, pretty: true);

vs

let s = j.to_str(pretty: true);

?

I want Json to behave like in JavaScript, where you could do things like:

j.foo.bar

This means that Json can't have instance level API because it will conflict with the data structure.

[eladb]

@ekeren @staycoolcall911 please assign this to me to update the spec for the next sprint

[ekeren]

This means that Json can't have instance level API because it will conflict with the data structure.

🤯

I didn't think of it

[skyrpex]

When building cloudy, I created a JsonSerializable interface, including typed jsonEncode and jsonDecode methods.

You can do many things with it, and more now with the satisfies keyword from TypeScript:

Make sure value is JsonSerializable but also retain the type:

const value = 1 satifies JsonSerializable;
//    ^? number

Erase the type:

const value: JsonSerializable = "hello world";
//    ^? JsonSerializable

Define unknown JSON return types:

function myApiCall(): JsonSerializable {
  return fetch(...);
}

Require a value to be JSON serializable before sending an API request, for example:

function sendData(value: JsonSerializable) {
   return fetch(...);
}

sendData(1); // valid
sendData("1"); // valid
sendData(new Date()); // not valid

Encode to JSON (and maintain the type of the structure):

const json = jsonEncode(["hello", 123]);
//    ^? JsonEncoded<[string, number]>

const value = jsonDecode(json);
//    ^? [string, number]

In order to manipulate a JsonSerializable object, you must typecheck first:

declare const value: JsonSerializable;
if (typeof value === "number") {
  console.log(value + 1);
} else if (typeof value === "string") {
  console.log("length", value.length);
} // ...

But you can also use libraries such as zod to conveniently validate the structure.

With the tools above, I don't feel like I'd need anything else in a language.

[Chriscbr]

@skyrpex If I understand correctly, I think your example shows that the satisfies keyword in TypeScript validates that the left operand is a subtype of the right. For example, if we put aside literal types in TypeScript for a moment, then the first statement you listed shows that 1 is of type number, which is a subtype of JsonSerializable. Any method that expects a JsonSerializable will gladly accept a number to be passed to it.

I personally feel is also the semantics that makes sense for Json in Wing, but I'm trying to better understand where others disagree. Is it that it doesn't feel right? Or that it breaks down logically in some use cases?

From Wikipedia:

Early versions of JSON (such as specified by RFC 4627) required that a valid JSON text must consist of only an object or an array type, which could contain other types within them. This restriction was dropped in RFC 7158, where a JSON text was redefined as any serialized value.

My mental model the only operation that you can do on a JSON value that is guaranteed to work converting it to a string. You can also try to view it as a number, or string, or boolean, or array, or object, but any of those operations could fail.

@ekeren interested in your input

[staycoolcall911]

@hasanaburayyan - I think we can close this issue, since thanks to you we now have support for Json in Wing! 🎉 The rest of the p2 tasks mentioned in this issue all have corresponding github issues. Please reopen if anyone feels differently.