remko / dsbackups

Package to work with Google Cloud Datastore exports
MIT License
4 stars 0 forks source link

Parsing nested ENTITY_PROTO property #1

Closed klon closed 2 years ago

klon commented 2 years ago

First, thanks a lot for your work on reverse engineering the backup format!

I am trying to use this lib to parse our Firestore database backup but unfortunately I am unable to parse the nested ENTITY_PROTO properties. Do you have any idea what proto spec is used for the nested binary data that occurs when meaning is 19 (ENTITY_PROTO)?

remko commented 2 years ago

I assume this is a marshaled EntityProto message, which you can get by adding this code (untested)

var ent aepb.EntityProto
err := proto.Unmarshal([]byte(*v.StringValue), &ent)
if err != nil {
  return nil, err
}
value.ValueType = &pb.Value_EntityValue{EntityValue: convertAEEntityValueToEntity(ent)}

The convertAEEntityToEntity function requires some work that may include factoring out some of the conversion work already done to be able to share the code. It shouldn't be hard or much work, but I since I don't use nested entities myself (and therefore don't have any test cases), I haven't done it myself.

Edit: It could be that you don't need any of the above, and all you need to do is replace the entity is not supported placeholder by

  v, err := UnmarshalAEEntity([]byte(*v.StringValue))
  if err != nil {
    return nil, err
  }
  value.ValueType = &pb.Value_EntityValue{EntityValue: v}
klon commented 2 years ago

Thanks, yes I got it to work finally! I am doing this in NodeJS with protobufjs and I had to change the proto definition for stringValue to use bytes instead of string as that library was mangling the data.