Typescript types for dynamic 'fields' not as smart as they could be

[roninjin10]

At Looker we have some endpoints that include a parameter called "fields" that takes a comma separated string of fields to include in the response. Because of the dynamic nature of this, every field in typescript has an optional type. So if you include 'last_name' for example, last_name is of type string|undefined because typescript is not smart enough to know you included it in fields.

With typescript generics, however, we can easily make typescript smart enough to include only the fields provided in the response type.

The typescript types for the functions generated by the code gen should wrap itself in a WithFields type that works exactly like the proof of concept but handles both Arrays and non arrays. If we want to make it generic, we can also make it so we specify the parameter 'fields' in the type.

[jkaster]

This will be a change to @looker/sdk-rtl and possibly @looker/sdk-codegen as well.

[joeldodge79]

My understanding was that the last_name?: string optional typing was based on the swagger/oas spec's required attribute and that we handled sparse responses (from fields params) using the Partial utility? (which is maybe effectively the same thing?)

Should we have a discussion about the higher level pattern as it might apply to other language sdks?

[roninjin10]

Yea this specifically replaces the use of Partial to make fields optional. Typescript has a robust type system that allows it to infer what fields will and will not be there based on the strings passed into 'fields'. I don't believe most other languages are as expressive with their type system to be able to do this.

[roninjin10]

To be more concrete, right now when we pass in fields like this

'id,last_name'

Id will be optional (even though it's always there) and last_name will be optional (even though it's always there. Additionally, first_name will be optional (even though it's never there).

With typescript we can make these types smart enough to know last_name and id are not optional and first_name is not included.

[joeldodge79]

For lack of a better place I'd like to document our interpretation of required/optional properties on schema objects WRT input vs output.

Context: For many of our endpoints we re-use the same exact Schema Object for input and output. An example is our User object:

GET /users / all_users(fields) -> User[] POST /users / create_user(body: User, fields) -> User

In this case the very same exact schema object User is referenced as the response to all_users (as an array) as well as the response to create_user AND as the input to create_user. The User schema object has NO required fields. Therefore, every field is "optional" per the spec definition.

For the purposes of translating this into strongly typed language SDKs we interpret required/optional properties differently depending on whether the object is a response or input:

As input/body-param a required property must correspond to the key being present in the inbound json payload (regardless of the key's value). Correct typing should reflect the property as required.

As output/response all properties defined in the schema will be included in the response payload. We are interpreting "required" (or lack thereof) on output/response objects as meaningless and instead assuming that all possible properties are required to be present in the output. Correct language typing should reflect this EXCEPT when a fields argument exists that applies an allowlist to a subset of properties. In that case all and only those properties in the subset must be required by the type system.

It appears that typescript can produce a wrapped response type to match our interpretation of required fields for response objects. We have not investigated if other language type systems can perform this function or not.