Deserialize Firebase documents

[BerndWessels]

Hi It seems that the serializers cannot deserialize Firebase document data.

This

firestore.collection("shops").snapshots().listen((QuerySnapshot snapshot) {
      Shop shop = serializers.deserializeWith<Shop>(Shop.serializer, snapshot.documents.first.data);
      print(snapshot.documents.first.data);
}

fails with

E/flutter ( 2658): [ERROR:topaz/lib/tonic/logging/dart_error.cc(16)] Unhandled exception:
E/flutter ( 2658): type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Iterable<dynamic>' in type cast where
E/flutter ( 2658):   _InternalLinkedHashMap is from dart:collection
E/flutter ( 2658):   String is from dart:core
E/flutter ( 2658):   Iterable is from dart:core

Is there a nice way to deserialize Firebase documents and collections without manually repeating all fields over and over again?

[davidmorgan]

That sounds like a mismatch between the JSON and your class. Would you mind posting details of both please?

There is an issue open to improve error messages on failed serialization, I hope to get to this soon:

https://github.com/google/built_value.dart/issues/408

[BerndWessels]

@davidmorgan Hi

this is the shop value:

library shop;

import 'package:built_value/built_value.dart';
import 'package:built_value/serializer.dart';
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:skipq_consumer_mobile_app/uuid.dart';

part 'shop.g.dart';

abstract class Shop implements Built<Shop, ShopBuilder> {
  static Serializer<Shop> get serializer => _$shopSerializer;

  String get id;

  String get name;

  @nullable
  GeoPoint get location;

  Shop._();

  factory Shop(String name) {
    return new _$Shop._(
      id: new Uuid().generateV4(),
      name: name,
    );
  }

  factory Shop.builder([updates(ShopBuilder b)]) {
    final builder = new ShopBuilder()
      ..id = new Uuid().generateV4()
      ..update(updates);

    return builder.build();
  }
}

and from firestore the document looks like this: String name, GeoPoint location

print(snapshot.documents.first.data);

I/flutter (20390): {name: shop 1b, location: Instance of 'GeoPoint'}

still getting the error:

E/flutter (20390): [ERROR:topaz/lib/tonic/logging/dart_error.cc(16)] Unhandled exception:
E/flutter (20390): type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Iterable<dynamic>' in type cast where
E/flutter (20390):   _InternalLinkedHashMap is from dart:collection
E/flutter (20390):   String is from dart:core
E/flutter (20390):   Iterable is from dart:core
E/flutter (20390): 
E/flutter (20390): #0      Object._as (dart:core/runtime/libobject_patch.dart:67:25)
E/flutter (20390): #1      BuiltJsonSerializers._deserialize (package:built_value/src/built_json_serializers.dart:139:52)
E/flutter (20390): #2      BuiltJsonSerializers.deserialize (package:built_value/src/built_json_serializers.dart:102:18)
E/flutter (20390): #3      BuiltJsonSerializers.deserializeWith (package:built_value/src/built_json_serializers.dart:32:12)
E/flutter (20390): #4      createConnectDataSource.<anonymous closure>.<anonymous closure> (package:skipq_consumer_mobile_app/middleware/middleware.dart:23:31)
E/flutter (20390): #5      _RootZone.runUnaryGuarded (dart:async/zone.dart:1316:10)
E/flutter (20390): #6      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:330:11)
E/flutter (20390): #7      _DelayedData.perform (dart:async/stream_impl.dart:578:14)
E/flutter (20390): #8      _StreamImplEvents.handleNext (dart:async/stream_impl.dart:694:11)
E/flutter (20390): #9      _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:654:7)
E/flutter (20390): #10     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
E/flutter (20390): #11     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)

[BerndWessels]

@davidmorgan I think the clue is somewhere in here type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Iterable<dynamic>' in type cast

It looks like the deserializer can only handle Iterable<dynamic> but firebase document data is a map.

To be honest I have no idea how built_value serializer / deserializer are supposed to work, the documentation that I could find didn't help a lot.

How would you deserialize documents and collections from firebase to built_value and built_collection ?

[davidmorgan]

Ah, it looks like you need StandardJsonPlugin. That switches to expecting a Map:

https://github.com/google/built_value.dart/blob/master/example/lib/example.dart#L85

But, it also looks like the data you're getting back is partially deserialized already ... what is a GeoPoint?

[BerndWessels]

The geopoint class comes from the firestore import and is basically latitude and longitude combined as this is a data type in firestore.

Firestore document data is a Map, I have no idea how JSON is represented in dart and how it relates to firebase.

[davidmorgan]

You'll definitely need StandardJsonPlugin as shown in the example:

final standardSerializers =
      (serializers.toBuilder()..addPlugin(new StandardJsonPlugin())).build();

although it refers to json, in fact what it does is to switch to using a Map. (This is 'standard JSON' because usually serialized JSON uses a map).

We will need to additionally account for classes like GeoPoint. Something like this should do it:

class GeoPointSerializer implements PrimitiveSerializer<GeoPoint> {
  final bool structured = false;
  @override
  final Iterable<Type> types = new BuiltList<Type>([GeoPoint]);
  @override
  final String wireName = 'GeoPoint';

  @override
  Object serialize(Serializers serializers, GeoPoint geoPoint,
      {FullType specifiedType: FullType.unspecified}) {
    return geoPoint;
  }

  @override
  GeoPoint deserialize(Serializers serializers, Object serialized,
      {FullType specifiedType: FullType.unspecified}) {
    return serialized as GeoPoint;
  }
}

And you'll need to add this serializer to serializers, in addition to StandardJsonPlugin:

final mySerializers =
    (serializers.toBuilder()
        ..add(new GeoPointSerializer()
        ..addPlugin(new StandardJsonPlugin())
    ).build();

If this works for you then I can look at supporting this use case better in built_value itself.

[BerndWessels]

Awesome, thank you. I will try it right away tomorrow morning (New Zealand) and post the results here.

[BerndWessels]

@davidmorgan thanks that worked, but now I realized another problem:

The serializer does not deserialize the document ID since it is not a field in the document in firestore.

So how do I keep the relation between the built_value Shop and the firestore document Shop ?

Can the serializer somehow take the document ID from and to the built_value ?

What are the implications for collections ?

This works but feels a bit to heavy:

    firestore.collection("shops").snapshots().listen((QuerySnapshot snapshot) {
      BuiltList<Shop> shops =
          new BuiltList<Shop>(snapshot.documents.map<Shop>((document) {
        var dataWithID = new Map.from(document.data)
          ..addEntries([new MapEntry("id", document.documentID)]);
        Shop shop = standardSerializers.deserializeWith<Shop>(
            Shop.serializer, dataWithID);
        print(shop);
        return shop;
      }).toList());

And I have not the slightest idea on how to deal with nested collections and their document IDs.

[davidmorgan]

Glad it worked :)

The code you have seems like one reasonable approach. I think you could do this in a general way, rather than needing to write it for every entity type, so there won't be too much boilerplate. i.e. you can write a method which takes a document and Shop.serializer and returns a Shop with an ID.

A second approach would be to mark the ID field @nullable. Then you will deserialize without the ID and can add it later.

Finally a third approach would be to store the ID separately, e.g. to put the Shop objects in a Map<Id, Shop>, or you could have a generic class Data<T> which stores an ID and a T.

I think what will work best mostly depends on how you end up using them.

[BerndWessels]

Thanks, I'll already took the first approach and that works fine for now. Let's close this until the next thing pops up 😉

[davidmorgan]

Sounds good. Thanks :)

[yehudamakarov]

Glad it worked :)

The code you have seems like one reasonable approach. I think you could do this in a general way, rather than needing to write it for every entity type, so there won't be too much boilerplate. i.e. you can write a method which takes a document and Shop.serializer and returns a Shop with an ID.

A second approach would be to mark the ID field @nullable. Then you will deserialize without the ID and can add it later.

Finally a third approach would be to store the ID separately, e.g. to put the Shop objects in a Map<Id, Shop>, or you could have a generic class Data<T> which stores an ID and a T.

I think what will work best mostly depends on how you end up using them.

Hey just seeing this. I'm working with firestore for the first time and realizing that the ConvertTo<>() method doesn't deserialize the ID. Which is super strange to me. Is there a paradigm I'm missing where you aren't supposed to be keeping an ID property on the model in the code? What I'm doing now is getting a snapshot, and then converting it to my model, and then manually adding the snapshot's ID onto my model? seems super weird unless I'm missing something.

What are your thoughts? right now I'm asking myself "why do you need an ID property on your data model?"

[davidmorgan]

I'm not familiar with firestore ... what is the ConvertTo method?

Might be better to ask someone who knows firestore :)

[RaimundWege]

In case of someone is trying this for GeoFirePoint of the geoflutterfire package:

import 'package:built_value/serializer.dart';
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:geoflutterfire/geoflutterfire.dart';

class GeoFirePointSerializer implements StructuredSerializer<GeoFirePoint> {
  final bool structured = false;
  @override
  final Iterable<Type> types = const [GeoFirePoint];
  @override
  final String wireName = 'GeoFirePoint';

  @override
  Iterable serialize(
    Serializers serializers,
    GeoFirePoint geoFirePoint, {
    FullType specifiedType = FullType.unspecified,
  }) =>
      ['geopoint', geoFirePoint.geoPoint, 'geohash', geoFirePoint.hash];

  @override
  GeoFirePoint deserialize(
    Serializers serializers,
    Iterable serialized, {
    FullType specifiedType = FullType.unspecified,
  }) {
    final iterator = serialized.iterator;
    while (iterator.moveNext()) {
      final key = iterator.current as String;
      iterator.moveNext();
      final value = iterator.current;
      switch (key) {
        case 'geopoint':
          if (value is GeoPoint) {
            return GeoFirePoint(value.latitude, value.longitude);
          }
      }
    }
    return null;
  }
}