RFC: Mapping Struct fields to Json fields

[hasanaburayyan]

Feature Spec

Wing now offers a way to define mappings between struct fields and their json field equivalent(s)

struct Person {
    #[json(name=["first_name", "fname"])]
    firstName: str;
    #[json(name=["last_name", "lname"])]
    lastName: str;
    age: num;
}

This allows an instance of person to be created from an of the following Json objects:

let a = {first_name: "foo", last_name: "bar", age: 21};
let b = {fname: "foo", lname: "bar", age: 21};
let c = {first_name: "foo", lname: "bar", age: 21};
...

The resulting schema uses "anyOf" to ensure validation

Rational

Currently in Wing when we define a Struct this generates a Json schema that can be used for runtime validation of json -> struct conversions. For example the following Struct:

struct Person {
    firstName: str;
    lastName: str;
    age: num;
}

Produces has a json schema defined as:

{
    id: "/Person",
    type: "object",
    properties: {
      firstName: { type: "string" },
      lastName: { type: "string" },
      age: { type: "number" },
    },
    required: [
      "firstName",
      "lastName",
      "age",
    ]
}

which means that a Json object that structurally fulfills the schema will be valid when attempting to convert to an instance of Person.

However, it will be the case that a Json object may have field names that mismatch the Wing struct definition. Its not a guarantee that all data read from the internet will come in the same case convention as Wing.

Usecases

1. mapping different data formats to struct
2. serialize and deserialize structs
3. Anything really its a comment that lives with the struct field

Alternate approaches

Custom Transformer

Add an optional argument to the toJson and fromJson methods that allow a custom transformer to be passed in.

struct Person {
  firstName: str;
  lastName: str;
  age: num;
}

let myTransformer = (obj: Json): Person => {
  let person =  Person {
    firstName: obj["first_name"].asStr(),
    lastName: obj["last_name"].asStr(),
    age: obj["age"].asNum(),
  };
  return person;
};

let somePersonObject = {};

let person = Person.fromJson(somePersonObject, myTransformer)

Overriding fromJson and toJson methods

struct Person {
  firstName: str;
  lastName: str;
  age: num;

  fromJson(obj: Json): Person => {
    let person =  Person {
      firstName: obj["first_name"].asStr(),
      lastName: obj["last_name"].asStr(),
      age: obj["age"].asNum(),
    };
    return person;  
  }

  toJson(self): Json => {
    return {
      "fist_name": self.firstName,
      "last_name": self.lastName,
      "age": self.age
    };
  }
}

---

How do other languages do this?

Golang (uses struct tags):

type Person struct {
  FirstName string `json:"first_name"`
  LastName  string `json:"last_name"`
  Age       int
}

Rust (serde):

#[derive(Serialize, Deserialize)]
pub struct Person {
  #[serde(rename = "first_name")]
  first_name: String,

  #[serde(rename = "last_name")]
  last_name: String,

  age: i32,
}

Java (Jackson):

public class Person {
  @JsonProperty("first_name")
  private String firstName;

  @JsonProperty("last_name")
  private String lastName;

  private int age;
}

C# (Newtonsoft.Json):

public class Person {
  [JsonProperty("first_name")]
  public string FirstName { get; set; }

  [JsonProperty("last_name")]
  public string LastName { get; set; }

  public int Age { get; set; }
}

[eladb]

1. Wing shouldn't do any automatic case conversion (just making sure)
2. I think we will add support for annotating any element, not just struct fields. I really don't like Go's syntax...
3. I am not a huge fan of using annotations for language features. Annotations should be used by userland libraries.

All that said, the specific use case of giving instructions to the json parser is something we should explicitly. If we have a list of the list of requirements and things we want to let users control, we can come up with the right language design and/or apis for that.

I recommend to repurpose this issue to something like "json parsing customization" or something like that and continue from there.

As for other types of serialization formats, that's a whole topic we need to cover separately with an RFC.

[skyrpex]

I don't like that syntax either... Doesn't feel extendable and IDE friendly (auto completion, etc). A system like that would allow de/ser very simple structures, but what about more complex ones?

Structures that contain valid JSON fields should be able to automatically be serialized/deserialized, right? In the case of Person, the JSON could be {"firstName": "Cristian","lastName": "Pallarés","phone": "777777777"}. Structures that contain non-json-serializable types would require some sort of implementation. I like the way you implement things in Rust, eg:

struct Person {
    firstName: str;
    lastName: str;
    phone: str;
}

impl JsonSerialize for Person {
    serialize(person: Person) {
        let json = new Json();
        json.add("first_name", person.firstName);
        json.add("last_name", person.lastName);
        json.add("phone_number", person.phone);
        return json;
    }
}

The Wing compiler can associate JsonSerialize.serialize with Person and use it whenever its needed.

[hasanaburayyan]

Wing shouldn't do any automatic case conversion (just making sure)

Oh yea for sure no sort of automatic case conversions, maybe I could have been more explicit in the example by mapping firstName to something like fname

As for other types of serialization formats, that's a whole topic we need to cover separately with an RFC.

I agree, that was not really the topic I wanted to convey in this RFC, I was more interested in discussing this custom parsing and how to make something thats extensible. Ill clean up the RFC to be more specific.

I really don't like Go's syntax...

[MarkMcCulloh]

I am not a huge fan of using annotations for language features. Annotations should be used by userland libraries.

@eladb You've mentioned this before and I'm curious where your feelings on this come from. Most languages I can think of that have annotations/decorators use it both for built-in (std lib) features as well as providing a way for users to create their own. I'm a fan of this personally (not the golang syntax though, yikes)

[Chriscbr]

I also prefer avoiding annotations/attributes for key language features. One reason is that annotations/attributes ought to be usable in userland, and supporting this would require designing a reflection API of some kind (something that might be better to defer until we have a library ecosystem and we've collected more use cases).

There are also several examples where annotations have been used in popular languages in places where a dedicated piece of syntax would be preferable. An obvious one that comes to mind is Java's @Override tag, which arguably should have been a dedicated keyword if it weren't for backwards compatibility. Some other examples that seem out of place to me are Python's @staticmethod, @classmethod, and @abstractmethod annotations, and C#'s [NotNull] and [DoesNotReturn] attributes.

This isn't to say that every feature should have a dedicated syntax; there will be plenty of use cases in "tail" of language usage. But something like how a struct is JSON serialized/deserialized feels like it could be important enough to have its own syntax to me, given the place Json has in Wing today.

[Chriscbr]

To play the other side, I'm also not sure how many use cases a dedicated syntax would need to support. Maybe listing them out (and seeing which can already be achieved in Wing today) would helps us figure out if field tags or annotations or some other system is the right call after all. Some examples:

1. As a user, I want to deserialize a JSON value with a field named "First Name" into a Wing struct with a field "firstName".
2. As a user, I want to take a Wing struct with a field "firstName" and serialize it to JSON with the field name "First Name"
3. As a user, I want to deserialize a JSON value expecting an enum-typed field named "priority", and convert it into a Wing struct with a field "name" that contains an enum of three priorities
4. As a user, I want to deserialize a JSON value expecting a string-typed field named "name", and convert it into a Wing struct, using a default string of "Anonymous" if the JSON value was missing the expected field

I don't think there's a way to do (1) today in Wing (because the field has a space in its name) but the others might have workarounds?

[hasanaburayyan]

@MarkMcCulloh @Chriscbr @eladb @skyrpex

I updated this RFC to be more specific to Json conversion. I also suggested another syntax closer to Rust's Serde (since everyone hates Go 😂)

I also added in examples of how other languages/libraries handle this problem. Im super open to the idea of adding some sort of more dedicated way to pass instructions to the parser/validator. Just fresh out of suggestions at the moment but Ill ponder more over the weekend.

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 90 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

[Chriscbr]

Thanks for updating the RFC Hasan! thought I'd share some general thoughts / questions

Putting aside the exact syntax, one way to view the options is by asking what we're exposing to the user to customize or override. (Are we giving them control of a schema? Or are we giving them control of a set of parsing or stringifying functions? Or are we giving them control of both, or something in between?)

Like you mentioned, in our current implementation, when a user runs MyStruct.fromJson(data), the compiler generates a JSON Schemas for the struct, and runs a JSON Schema validator to check "data" is valid. It doesn't do anything to transform the fields.

Suppose in order to support custom struct fields we added some syntax that lets you change the schema in some way. I'll borrow the syntax from the original issue:

struct Person {
    #[json(name=["first_name", "fname"])]
    firstName: str;
    #[json(name=["last_name", "lname"])]
    lastName: str;
    age: num;
}

Now, if you tried parsing the object, an updated schema would be used, so the "first_name" field would be validated as satisfying the schema. However, if the data isn't automatically transformed in some way, you wouldn't be able to access it 😱:

let person = Person.fromJson(Json { first_name: "Jeff", last_name: "Bezos", age: 60 });
log(person.firstName); // nil

For serialization into JSON, a similar issue could happen. To fix it, we'd need to generate code at compile time that transforms the "firstName" field into "first_name".

---

The other note I want to add is that I think renaming fields is just one example of a custom mapping / serialization option. Some other use cases might be:

• When converting from JSON, you may want to inject a default value if the original data has a null/empty value for a field
• When converting from JSON, you might want to parse a string as a number or as a duration or as a datetime or as an enum
• When converting to JSON, you might want to convert a number or duration or datetime into a string
• When converting from JSON, you may want to reject an object if it has unknown fields, or you may want to simply remove those unknown fields
• When converting to/from JSON, you may want to validate that string matches a certain regex, or a number is within a certain range
• When converting from JSON, you may want to validate that if field A is present or if field A has a certain value, then field B must be present or field B doesn't take on a certain value

I don't know if we necessarily need a solution that solves all of these all at once. But we might want to consider "what kinds of solutions would allow us to support these kinds of customizations generally / without requiring language support for each one"?

[Chriscbr]

BTW - one other hybrid syntax could be to add some annotation to the struct that references the transformer(s) in some way:

@serializer(PersonSerializer)
// or maybe @serializer(new PersonSerializer())
struct Person {
  firstName: str;
  lastName: str;
  age: num;
}

class PersonSerializer {
  fromJson(obj: Json): Person => {
    let person =  Person {
      firstName: obj["first_name"].asStr(),
      lastName: obj["last_name"].asStr(),
      age: obj["age"].asNum(),
    };
    return person;  
  }

  toJson(self): Json => {
    return {
      "fist_name": self.firstName,
      "last_name": self.lastName,
      "age": self.age
    };
  }
}

(I'm not sure if I like this better than any of the other options - but wanted to add it for comparison)