[PROJECT] New proto-backed Firestore structure

jcqli commented 5 months ago

Parent issue tracking the work needed to introduce protos as our db and Firestore schema.

[x] Implement proto converters for Android (https://github.com/google/ground-android/issues/2049)
[x] #1933
[x] #1934
[x] https://github.com/google/ground-platform/issues/1859
[x] https://github.com/google/ground-platform/issues/1775
[x] https://github.com/google/ground-android/issues/2500
[x] https://github.com/google/ground-android/issues/2501
[x] #1778
[x] https://github.com/google/ground-platform/issues/1857
[x] https://github.com/google/ground-android/issues/2498
[x] https://github.com/google/ground-android/issues/2499
[x] https://github.com/google/ground-android/issues/2494
[x] https://github.com/google/ground-android/issues/2548
[x] #1779
[x] #1780
[x] #1781
[x] https://github.com/google/ground-android/issues/2468
[ ] https://github.com/google/ground-platform/issues/1858

Additional tasks related to this migration:

gino-m commented 4 months ago

Did some research on various protoc plugins for generating js+ts code from protodef. We need something which makes metadata accessible (esp. field numbers) in order to implement the representation described in #2455. Here are my findings:

protobufjs ✅

The most popular lib I found (16M weekly downloads), originally designed by Google. It can generate js and ts, but to get what we want you need to call it three times - once to generate the js, once to generate to ts bindings, and once to generate JSON with pb metadata.

npm i -D protobufjs protobufjs-cli
npx pbjs -t static-module ../proto/survey.proto -o dist/survey.js
npx pbts dist/survey.js -o dist/survey.d.ts
npx pbjs -t json ../proto/survey.proto -o dist/survey.proto.json

We could load the JSON at runtime to pick out field numbers we need.

Alternatives considered

@protobuf-ts/plugin

146k downloads/week. Generates pure ts code in a single command, and includes field numbers in the message metadata. But given this is a less "standard", less used library, prefer using protobufjs.

ts-proto

OTOO 300k weekly downloads, and doesn't encode the field numbers anywhere we can use at runtime, so we can exclude this one for now.

protoc-gen-ts

It seems the protoc packaged with the one didn't support my M1 Mac.

npm i -D @protobuf-ts/plugin
npx protoc --ts_out dist --proto_path ../proto survey.proto loi.proto submission.proto

It doesn't produce or consume JSON, but it's fairly easy to go from ts object<>JSON anyway, plus we'll prob want more control over the formats.

grpc_tools_node_protoc_ts

Only 100k downloads per week, but uses the standard google grpc-tools. Doesn't seem to embed the set of valid field numbers, though individual fields can be fetched by field number:

npm i -D grpc-tools grpc_tools_node_protoc_ts
npx grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./dist --grpc_out=grpc_js:./dist --plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin -I ../proto survey.proto
npx protoc --plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts --ts_out=grpc_js:./dist -I ../proto survey.proto

gino-m commented 4 months ago

Also, carrying forward proposal from https://github.com/google/ground-android/issues/2455:

Create lightweight converters in Kotlin and TypeScript which take a Map<Long, Any> and produce protobufs and vice-versa, where keys in Firestore correspond to a protobuf field number, and values primitive types supported by proto.

Example survey data before:

{
  "title": "My survey",
  "description": "An example survey",
   //...
}

Example survey data after:

{
  1: "My survey",
  2: "An example survey",
  ...
}

This has several advantages:

Generated code ensures a consistent schema across web, Android, and Cloud Functions.
Using field IDs instead of names allows field names can be changes without breaking clients.
Storaging values as Firestore values rather than binary allows Firestore query filters to work without duplicating values in "index" fields, and it allows data to be viewed and debugged using standard tools.
The implementation is simple, easy to write and debug.
Using field numbers instead of names allows this sol'n to work with Android "lite" protobufs in which name bindings aren't available at runtime.
Writing directly to generated code

This has a few disadvantages as well:

Maintaining backwards compatibility with existing test surveys is very difficult. We would probably want to zap the dbs and start from scratch.
Indirection in key names in Firestore; would need to check protobuf def to understand which field is which.

jcqli commented 1 month ago

Filed https://github.com/google/ground-platform/issues/1966 to track cleanup of old code @gino-m shall we mark this as closed?

gino-m commented 1 month ago

Yes, I think we can!

google / ground-platform