cedvdb commented 3 years ago

Dart should have a data type that allows to work with:

Typed data
Partial data

What dart has at the moment is not typed (maps). I propose the use of python TypedDictionnary coupled with utilities a la TypeScript.

use case

Consider this use case: You are on a Product page and you want to update certain properties one by one. One way of implementing it in typescript would be:

class ProductAPI {
  update(String id, Partial<Product> update) {
     firestore.collection('products').doc(id).update(updates);
  }
}
// ...
productAPI.update(id, {favorite: true });

In dart, you'd have to use a map in place of Partial which is not typed.

Typescript utilities

Typescript has a set of utility types that are extremely useful, especially for front end development. In the front end often you are not dealing with a whole entity but a partial one.

Here are some of the utilities (full list)

Partial<Type> : Constructs a type with all properties of Type set to optional. This utility will return a type that represents all subsets of a given type.

Required<Type>: Constructs a type consisting of all properties of Type set to required. The opposite of Partial.

Readonly<Type> : Constructs a type with all properties of Type set to readonly, meaning the properties of the constructed type cannot be reassigned.

Pick<Type, Keys>: Constructs a type by picking the set of properties Keys (string literal or union of string literals) from Type.

and more

typed dictionnaries

While I don't use python, it seems they have something that could solve this, and other issues like json serialization.

Typed dictionnaries : This PEP proposes a type constructor typing. TypedDict to support the use case where a dictionary object has a specific set of string keys, each with a value of a specific type.

Representing an object or structured data using (potentially nested) dictionaries with string keys (instead of a user-defined class) is a common pattern in Python programs. Representing JSON objects is perhaps the canonical use case, and this is popular enough that Python ships with a JSON library. This PEP proposes a way to allow such code to be type checked more effectively.

Something like that in dart would be fantastic where you'd also allow typescript modifiers. The original use case could be typed:

typedDictionnary Product {
   final String name;
   final bool favorite;
}

class ProductAPI {
   // ...
   update(String id, Partial<Product> updates) {
      firebase.doc(id).set( updates.toJson());
   }
}

productAPI. update({ id, favorite: true });

Static Meta programing:

While meta programing would supposedly solve the issue of data classes I'm not sure it can handle the Partial part.

mateusfccp commented 3 years ago

Related: #783.

Levi-Lesches commented 3 years ago

typedDictionnary Product {
  final String name;
  final bool favorite;
}

class ProductAPI {
  update(String id, Partial<Product> updates) {
    firebase.doc(id).set( updates.toJson());
  }
}

What are the advantages of that over the following?

class Product {
  final String? name;  // nullable because of how I'm interpreting "partial"
  final bool? favorite;
  const Product({this.name, this.favorite});
  // Map<String, dynamic> toJson() => { ... };
}

class ProductAPI {
  Future<void> update(String id, Product updates) => 
    firebase.collection("products").doc(id).set( updates.toJson() );
}

Is it the automatic toJson? In other words, by partial, do you mean "all fields are nullable"?

cedvdb commented 3 years ago

In your code above every property of the product you specified is nullable it does not correctly reflect a Product data, where some fields are not nullable. You could have default values on your database and be sure that when you read a product those fields are never null. However when you update a product, you often want to update a single property. Typescript keyword resolve this.

That's the advantage of the utility keywords and how they'd affect a typedDic. Another feature this has is that the properties are string so you can easily read them, which makes it easy to serialize / deserialize.

Levi-Lesches commented 3 years ago

It seems you're looking for mutable fields, not final. Instead of creating a new "partial" Product object and merging the two, you would update the existing object one property at a time in your UI or other logic (and then serialize it to the database when you're done).

class Product {
  String name;
  bool isFavorite;
  Product({required this.name, required this.favorite});
}

// in your UI
void toggleFavorite(Product product) => setState(
  () => product.isFavorite = !product.isFavorite
);

// if you have other logic
Future<void> updateProperties(Product product) async =>
  product.isFavorite = await isFavorite(product);

A copyWith method could also help. They traditionally have optional parameters for all your fields, that way you can update specific fields of a class without creating a new class or object. For example:

class Product {
  final String name;
  final bool isFavorite;
  const Product({required this.name, required this.isFavorite});

  Product copyWith({
    String? name,
    bool? isFavorite,
  }) => Product(
    name: name ?? this.name,
    isFavorite: isFavorite ?? this.isFavorite,
  );
}

// in your UI
void toggleFavorite(MyData dataModel) => 
  dataModel.product = dataModel.product.copyWith(isFavorite: !product.isFavorite);

As you can see, copyWith is a longer way of just making your fields mutable (ie, not making them final), and regular mutable fields are better for most circumstances.

If you're looking for a special class that already has copyWith included, see #314. For built-in code-generation that can make a copyWith for you (without forcing you to use a "partial" or "data-only" class), see #1482, and my proposal at #1565.

cedvdb commented 3 years ago

It seems you're looking for mutable fields, not final.

Absolutely not. This is going a bit on a tangent but I'll try to explain more clearly.

I want :

a non mutable Product with final fields with some of them non-nullable. That's the product I read from DB / cache. Let's call it class ProductRead.
a way of representing that same data model partially in a statically typed fashion only some of the properties. This might be a runtime type though.

  // in your UI
void toggleFavorite(MyData dataModel) => 
  dataModel.product = dataModel.product.copyWith(isFavorite: !product.isFavorite);

In a real application you would also update the favorite property on the database. You would not just pass the whole product. so you'd have this:

update(Map<String, dynamic> updates); // not typed !

Surely you can see the need of such mechanism for front end applications where updating one property at a time is a thing. Currently there is no way of doing that and it is a inconvenience for me. I'm currently using maps in place I would like the compiler to scream at me when I rename a property. This is really not fun.

Now I hope your static meta programing proposal could change that but I'm not sure it could do something like the feature I'm proposing here.

Another examples of such shortcoming of the language that could be resolved by this:

https://pub.dev/packages/reactive_forms#what-is-not-reactive-forms the most used form library in flutter that cannot type its output. Dart short comings would make the public API unusable and far from its original goal of being close to the angular version.

Whatever the answer to this is, be it meta programing or something like the feature I propose, it cannot come soon enough as I'm annoyed every day by this.

Levi-Lesches commented 3 years ago

It seems you're looking for mutable fields, not final.

Absolutely not.

I mean, it seems like you are. "Malleable" means changeable, and "updating a single property" means you want to be able to modify its value. I know Dart and Flutter sometimes make mutable data look ancient and obsolete but they're often legitimate solutions. Unless you have a super compelling reason not to (which I'd be glad to advise on), this would work best for your case.

As for metaprogramming, that would be relevant here in one of two ways:

automatically create a parallel class with all-nullable fields
automatically create a copyWith method and keep it up-to-date

cedvdb commented 3 years ago

"Malleable" means changeable, and "updating a single property" means you want to be able to modify its value.

Then I'm choosing my words poorly and I edited the title. My product is never updated on the front end, my data is not malleable. I'm updating the data on the backend and it's only the representation of the data that is malleable.

consider this:

class Product {
  final String name;
  final bool favorite;
  // + 100 more properties
}

class ProductsAPI {
   // stream of products, changes on the backend are reflected here
   final products$ = firestore.collection('products').snapshots();

   update(Map<String, dynamic> updates, String id) {
     // updates the product on the backend
     return firestore.collection('products').doc(id).update(updates);
   }

}

How do you propose I achieve this with a correctly typed update (not using a map) that would allow me to update any combination of properties of a product ?

Levi-Lesches commented 3 years ago

So to clarify, you're getting fresh data from a backend but you don't want the app itself to change the data being reflected.

So in that case immutable would be a good way to go, but I would suggest that instead of storing the partial changes, you just override it with the complete object, and add a .fromJson to your class to get the new object. So:

class Product {
  final String name;
  final bool isFavorite;

  Product.fromJson(Map json) : 
    name = json ["name"],
    isFavorite = json ["isFavorite"];

  Map<String, dynamic> toJson() => {
    "name": name,
    "isFavorite": isFavorite,
  };
}

class UIData {
  // basic data structures for your app
  static Map<String, Product> products;  // maps names (or IDs) to products
}

class API {
  void listen() => firebase.collection("products").snapshots().listen(
    (QuerySnapshot snapshot) {
      for (final QueryDocumentSnapshot doc in snapshot.docs) {
        // get the data represented by this doc and save it to UIData
        final Map<String, dynamic> json = doc.data();
        final Product product = Product.fromJson(json);
        UIData.products [product.name] = product;
      }
    }
  );

  /// Updates a single product
  Future<void> updateProduct(Product product) => 
    firebase.collection("products").doc(product.name).set(product.toJson());
}

This is the pattern I use (with a bit more separation between data and firestore) and I find it works well. Hopefully it helps you too.

cedvdb commented 3 years ago

We are getting somewhere.

  Future<void> updateProduct(Product product) => 
    firebase.collection("products").doc(product.name).set(product.toJson());

How do you update only the favorite property with this code ? That's my point , I have to use this:

  Future<void> updateProduct(Map<String, dynamic> updates) => 
    firebase.collection("products").doc(product.name).set(updates);

but in typescript I can type that:

  Promise<void> updateProduct(Partial<Product> updates) => 
    firebase.collection("products").doc(product.name).set(updates);

Levi-Lesches commented 3 years ago

By updating the whole thing! If you keep the whole object in the database, you don't have to worry about applying specific changes -- just throw away the old object and take the new one. To show that this works, if you try it on Firestore and watch the console, only the property you change will turn orange. So even though you're replacing an entire object, practically speaking you only change what you need.

For example:

// you say the app doesn't make these changes, so pretend this is happening in the backend
Product a = Product(name: "toy", isFavorite: false);
Product b = Product(name: "toy", isFavorite: true);

await updateProduct(a);  // firestore/products/toy = {"name": "toy", "isFavorite": false}
await updateProduct(b);  // firestore/products/toy = {"name": "toy", "isFavorite": true}

And in your app, the stream listener gets called twice, one with each of the above values. Since Product.name is the same, your app knows these are the same product and updates UIData.products accordingly. So even though you're getting two different objects, from your apps perspective, you only changed one property, isFavorite.

cedvdb commented 3 years ago

By updating the whole thing!

Consider a Product that is updated by a team of people concurrently and you don't want to override other people changes with old values. You could still have the old version locally because the update didn't get to you and when you update by giving it the whole thing it overrides the change of someone else that made the change 1 second before you.

Honestly while I'm not against discussion I think some of the refutations about this are not honest. Updating the whole thing is more akin to a workaround, it's sending more bytes over the network than necessary. I can have a diff mechanism as well, that would work. Ultimately there are solutions, what I'm advocating for is sensible ones, a diff system in place of a partial keyword is not one of them.

Levi-Lesches commented 3 years ago

Product is updated by many people concurrently and you don't want to override other people changes with old values.

Okay so this part clarified it a lot for me. I played around a lot in DartPad and yeah, I see what you mean. At least in Firebase, you can use Transactions to download the latest data and use some Map logic to merge the two, but you're right that the only way to do it typed is to have a PartialProduct with all nullable fields. Glad you brought me 'round.

Metaprogramming can help here by automatically creating that class PartialX with all the fields of X just nullable, and with a toJson as well (essentially a Partial<Product in your terms). Having some built-in mechanism would be nice, but then again, if such code were built into a package that wouldn't be too bad either. It could look something like this (assuming the details from #1565 stick):

// product.dart
import "partial_classes.dart";  // the PartialClass annotation

part "generated:product.g.dart";  // the partial class itself

@PartialClass(mutable: false)
class Product {
  final String name;
  final bool isFavorite;

  const Product({
    required this.name,
    required this.isFavorite,
  });
}

And this gets generated:

// product.g.dart
part of "product.dart";

class PartialProduct {
  final String? name;
  final bool? isFavorite;

  const Product({this.name, this.isFavorite});

  Product merge(Product other) => Product(
    name: name ?? other.name,
    isFavorite: isFavorite ?? other.isFavorite,
  );
}

Then changes would be made to PartialProducts instead of regular Products, and they can be merged with the latest data whenever needed, either for the frontend or the backend.

Here would be the metaprogramming code according to #1565, if you're interested

```dart // partial_classes.dart import "dart:code_gen"; /// A macro that generates a new `PartialFoo` from a user-written class `Foo` class PartialClasses extends ClassMacro { final bool mutable; const PartialClasses({this.mutable = true}); /// this code gets passed to `dart format`, don't worry @override String generateTopLevel() => [ "class Partial$sourceName {", // declare nullable (and maybe mutable) fields for (final Variable field in fields) "${mutable ? 'final' : ''} ${field.type}? ${field.name}", // constructor "const Partial$sourceName({", for (final String field in fieldNames) "this.${field.name},", "});", // the merge function to merge with the original class "$sourceName merge($sourceName other) => $sourceName(", for (final String field in fieldNames) "$field: $field ?? other.$field,", ");", "}", ].join("\n"); } ```

The reason I keep touting metaprogramming everywhere is because I see a lot of issues where it's really a matter of boilerplate that no one wants to write. Metaprogramming can help by auto-generating these ideas for you so the Dart team doesn't have to manually implement every special case (of which there are a lot), or can at least take their time.

cedvdb commented 3 years ago

The thing about a PartialProduct generated by metaprogramming is that it's just a Product class with every field being nullable and it's not exactly the same as a Partial where the fields just don't exist at all.

If you give a PartialProduct to the above update functions you'd have to filter out null values which brings the new issue of: How do you update a field with a null value ?

I get that meta-programing would resolve a lot of things just by the nature of it being sort of a "language builder". However I don't think it can solve this, as the data here would be constructed at runtime (You don't know which property the user wants to update in advance). Also something directly integrated in the language gives an higher level of confidence than a third party library that uses meta programming. So things that are truly useful could be still integrated in the language directly.

I gotta say that your API looks nice tho

SandeepGamot commented 2 years ago

@cedvdb Did you figure out something eventually or had to resort to using Map?

cedvdb commented 2 years ago

I've been using maps, as the alternatives did not look attractive to me, the funny thing is that I saw the notification for your comment while going to this repo for checking the advances in static meta programming while thinking about this issue again because I just add an issue with a typo. So I'm still annoyed by this, weekly 😁.

After static metaprogramming I envision still using a map but having static metaprogramming generate the strings for the keys in the map (eg: Product.keys.name). To me that would be a sensible solution. Currently it's annoying that it's not caught by the compiler, but with proper unit tests you eventually catch those issues.

I'd gladly take the compiler catching those before tests though, simply because it's easier to debug than a failing test.

dart-lang / language

Partial typed dictionnaries #1635

use case

Typescript utilities

typed dictionnaries

Static Meta programing: