search

Azure / azure-sdk

This is the Azure SDK parent repository and mostly contains documentation around guidelines and policies as well as the releases for the various languages supported by the Azure SDK.
http://azure.github.io/azure-sdk
MIT License
482 stars 297 forks source link

Change Request: Optional type for C# #1566

Closed MiYanni closed 6 months ago

MiYanni commented 4 years ago

The Basics

Which design guideline is affected?

Any guidelines around type definition and usage relating to nullability, requiredness, and defaults.

Which target languages are affected?

Languages that do not have proper representation of an optional type, or languages currently not utilizing their optional type in SDKs.

Describe the change

The basic premise is to use an Optional-style type to represent server-side optionality. Meaning, if a particular value is not required to be sent (or received) from the service, it should be represented as an optional value. If the Optional type has no value, it represents when a value is omitted from being sent or recieved. Currently in C#, we use null to represent this scenario. However, null can cause issues which are outlined below.

Reasoning

For the sake of this issue, the primary use of Optional on the API surface would be the property for a type or as a parameter to a method. Properties of types (class or struct) usually represents some data that is to be sent to a service or recieved from a service. Similarly, service method parameters usually represent a piece of data being sent to the service. Therefore, these uses are what will be the focus here, although Optional itself may be used internally to the SDK as applicable.

This Optional type is used for clarification of client-service interaction and the data contract between them. It creates an established pattern for SDKs to use, and explicitly indicates to the user when a value will not be sent to the service or when a value was not recieved from the service. This creates stronger value (and null) intent for C# (or other languages). This is not meant to replace null, but to give null proper data representation and make Optional be the defacto omitted representation.

There are a number of aspects related to this topic, so I'll break them down as follows:

(Note: The section hyperlinks don't work when this document is rendered as an issue description)

Nullability

There are a few aspects to keep in mind when talking about nullability. For context, these are the C# nullability concepts:

In general for our Azure SDKs when a null value is provided/recieved for a type that interacts with the service, it is assumed that this value is omitted. This means that either the SDK will not send this value to the service (it isn't included in the request) or the SDK did not receive a value from the service (it wasn't present in the response). That is the basic/high-level idea of null from a service interaction viewpoint.

Service-Defined Null Value

In OpenAPI, a service can define if values are nullable. This means that the service will accept a null value in the data provided to it. This is in the form of x-nullable: true/false. For example, if the data being sent to the service is JSON, x-nullable: true would allow a JSON null to be sent instead of the value's type.

This creates a bit of confusion for us when considering C#'s null value. Because of our current use of null to mean omitted, how do we distinguish between when null should represent omitted or it should represent a data format's null? As it stands, we can't. That is one of the big problems presented in this issue. I explore this deeper in the Update/Patch Scenario section.

Collections and Dictionaries

Collections and key-value pair dictionaries can also contain types that may have null values. For example, a collection with 3 null entries is different than the collection itself being null. Currently, in the former, we would send an entity with the 3 null entries to the service, and in the latter, we would not send any collection. The recieving mechanics for these scenarios are the same, as null entries will be created in a collection for the former, and the latter will be a null collection entirely.

In the collection/dictionary scenario, the addition of an Optional type would not benefit the element type as there is not the idea of 'optional entries'. Either the entry exists in the collection or it does not, and the 'does not' represents omitted. However, for the collection itself, null could represent either the service omitted the collection or the collection was provided as null from the service.

Nesting an Optional type 'inside' a container type would not be applicable. For example, these would not apply:

We would want to be explicit about how we use Optional, since the purpose is to represent something that may or may not exist. If the type itself has a mechanism for this, like collections, Optional should not be used.

Nullable Reference Types

In C# 8, nullable reference types was introduced. This means that explicit declaration of nullability (via ?) is required for all types, value and reference types alike. This is required in usage of nullable reference types. However, reference types themselves still have a default value of null, which is further described in a section below.

This change to reference types means that explicit usage of nullability is required. Therefore, everything declared a null needs to have a purpose for being null. Optional helps this transition from standard reference types to nullable reference types. For example, if the type was Optional<string> before C# 8 and the null value of the string was unused (not a valid value), then it would remain Optional<string> with nullable references enabled. If Optional isn't being used, right now, reference types would need a ? added to them to indicate that null can be used, and means the value is not sent to the server. At that point, every reference type needs to be checked/investigated to see if null would be used as omitted or not. It will take additional effort for our SDK team, and additional understanding/documentation for SDK consumers.

Required and Default Values

For our SDKs, the services determine which values are necessary to send to make a valid request and a contract of the expected values we would recieve from the service as a response. Some values are required, meaning that they must be sent and the service or client will always expect them. This comes up most often for properties of request bodies, which become either parameters to a method or properties on a model in the SDK. The concept of a value being required or not required is optionality, which is the primary focus of this issue.

However, a value can still be optional even if it is required. This is in the form of defaults. The service definition (such as OpenAPI) can state two different kinds of defaults, server-side defaults and client-side defaults. Server-side defaults indicate that when a value is omitted from an operation, the service will set that value internally to a default value. Client-side defaults indicate an appropriate default value for a value to be sent to the server if the client is unsure of what to send.

When a service definition indicates a client-side default, this means clients can design their API surface to have that value be optional. If the SDK user does not provide a value, the SDK itself knows what value to send. Therefore, even if the value is marked as required, the SDK will always have a value in-hand to send. Thus, from a user's perspective, it will always be optional.

If a type Optional is created, it would be used when:

I have created a Gist a few months back that shows how null, required/optional, and client-side defaults would interact with each other if an Optional type was introduced. I give example operation method definitions that show how these three aspects interact with each other, and now the parameter changes given an Optional type exists. It allows each aspect to be explicitly represented as part of the parameter declaration.

Value-type Defaults

Value types have a significant difference over reference types (for C#). Their default value is a parameterless-constructed instance of the type, unlike reference types that use null. In most cases, we simply use a Nullable<T> for structs. Then, null would clearly represent omitted. However, this causes another issue. What does the default value for a struct represent?

In this scenario, default for reference types (which is null) will work as omitted while null for structs (as Nullable<T>) is omitted and default is not a valid value. The specifics of this issue with proposed implementations are shown in the issue @AlexanderSher wrote up on extensible enums. An extensible enum in his scenario is a struct.

For a real C# enum, there is additional confusion as an enums default is (usually) 0, which would be the first value defined in the enum. Services may (or may not) state explicitly that this value is the client-side default. Because of this, the design has been to make the enum as Nullable<T> so that default will appropriately represent a valid value.

Update/Patch Scenario

In terms of service interaction, a service could return null as a valid value. As mentioned above, for example, a service can contain a JSON null value in the data recieved from the client. Additionally, a client could send a JSON null and the service would understand it as a particular value. In this scenario, that null has a different significance than simply omitting a value.

A common scenario is to update a particular entity in a service. There are few forms of this, but for this particular aspect, let's focus on an HTTP PATCH scenario. For this, a service expects only the data that should be updated. Therefore, omitted values are a no-op on the entity's current data. This works just fine when you want to set data. But, there is a particular interaction that is not easily covered, when a client wants to unset a value. This would effectively 'clear' the value held by the service for that entity.

As it stands, the pattern used for this action is setting the entity's value to null. But this causes C# null on our client to represent two different values for this operation. If my understanding is correct, because of this duality, we currently do not support unset. Instead, any null values for these operations still only represent omitted.

With an Optional type, the unset action can be clearly represented. With it, C# null represents a null to/from the server. If a value (null or otherwise) is not provided to the Optional type, it will not be sent (or was not recived) to/from the server. This allows the entity update scenario to propertly unset values on the entity. From my understanding, Cosmos and Digital Twins are currently designing their update/merge APIs. This would be a great time to introduce the Optional concept as it would make their efforts easier.

However, there is an alternative but requires service support. If the service supports JSON Patch, then using null for unset is not necessary. @pakrym mentions this format in this issue here. Personally, I haven't seen this used before in conjunction with Azure services. Though, this solution is limited to only JSON data exchanges.

Implementations and Discussions

For an Optional type, there has been some notable discussions and implementations.

Discussions

Implementations

Other

MiYanni commented 4 years ago

Just a quick note: I realized I didn't mention this in the description. Without a type wrapper like Optional, properties don't have an easy way to represent their optionality in models that do not have explicit parameterized constructors (like straight DTO models). Even though you can set a default value for properties, that style of optionality isn't easily conveyed to users of that model. If you had a convention where optional parameters use Optional and non-optional parameters do not, the properties themselves would easily convey the difference between being required or optional. Things can get confusing when mixing this with client and service default values, so that would need to be discussed with the arch board.

github-actions[bot] commented 6 months ago

Hi @MiYanni, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.