FR: realtime event listeners for Realtime Database

[johndelavega]

Or provide some general info on how I may be able to code it myself.

Thanks John

[hiranya911]

You can use the streaming support available in the RTDB REST API to implement this: https://firebase.google.com/docs/reference/rest/database/#section-streaming

That's how we supported it in our Python SDK.

[johndelavega]

Looks like something I could do. You can assign it to me.

As you mentioned in issue#35 to do this incrementally, I could try to get something working with very basic support first then we can take it from there.

[hiranya911]

Thanks @johndelavega. I don't think our repos allow assigning issues to non-Googlers. But consider it assigned :)

As a first step, please outline what the public API of this feature would look like here. Once we settle on that, you can go ahead and try to implement it. I assume this API will return some sort of a channel that allows clients to receive and consume updates. It might not be a bad idea to align with the semantics of the Python API here. See the listen() method at https://firebase.google.com/docs/reference/admin/python/firebase_admin.db#reference

[johndelavega]

Here are my suggestion for public API and does not include a channel, just a callback function, but we can probably create one that does include a channel. Internal code will use goroutines for the async nature of a listener. The empty interface arg is similar to existing public Get/Set API.

func (r Ref) Listen(ctx context.Context, func(v interface{}) (Event, error) func (r *Event) Close() error

Ideally, we can call Listen from the same Ref multiple times with different callback functions, returning unique Event (or *EventListener) object used for closing http connection when done.

Need clarification for the use case of ListenerRegistration in Python API, is this optional?

[hiranya911]

The ListenerRegistration is equivalent to what you call EventListener.

Perhaps, a better way to do this in Go is to align with the Firestore listen API:

https://godoc.org/cloud.google.com/go/firestore#DocumentRef.Snapshots https://godoc.org/cloud.google.com/go/firestore#DocumentSnapshotIterator

When applied to RTDB, this would look like this:

func (r *Ref) Listen(ctx context.Context) (*SnapshotIterator, error)

func (i *SnapshotIterator) Next() (*Event, error)
func (i *SnapshotIterator) Stop()

type Event struct {
  Path string
  EventType int // this can be an enum type
}
func (e *Event) Unmarshal(v interface{}) error

Something along these lines? @bklimt do you have any suggestions?

[bklimt]

I really like having an Unmarshal method rather than having the callback take interface{}, because it means that consumers of the API (1) can use their own structs, and (2) don't have to guess the type of primitives (int vs int32, etc).

The trade-off between having a callback versus a SnapshotIterator is less obvious to me. I would ask @wilhuff if he has any advice.

[johndelavega]

I just want to point out that consistency between existing API such as Get and Set and the new Listen API is important. Most apps using Listen will likely be using Get, Set, and other existing APIs.

[wilhuff]

@jadekler Is the maintainer of the Firestore Go SDK and may have more context on the choice. As far as I can tell, this is a just a choice that's been made cloud-wide, and Firestore is merely being consistent with it:

https://cloud.google.com/spanner/docs/reference/libraries#client-libraries-usage-go

[hiranya911]

@johndelavega this API is not really going to work:

func (r *Ref) Listen(ctx context.Context, func(v interface{})) (*Event, error)

Note that unlike the existing Get() function (which is invoked by the developer), the callback function is invoked by the SDK. The developer will have to implement a bunch of conditionals with type assertions to properly handle different inputs:

callback := func(v interface{}) {
  switch v.(type) {
    case MyType:
      // handle MyType
    case MyOtherType:
     // handle MyOtherType

   // so on and on
  }
}

This also doesn't expose the event type (child_added, child_removed, value_changed etc) or the child path where the event it originating from. Remember that listening to /foo will yield events whenever /foo or any of its children change.

[hiranya911]

@bklimt I think my previous comment also explains why a callback API is not adequate. At least not without a wrapper type as follows:

func (r *Ref) Listen(ctx context.Context, func(e *Event)) (*ListenerRegistration, error)

type Event struct {
  Path string
  EventType int // this can be an enum type
}
func (e *Event) Unmarshal(v interface{}) error

[johndelavega]

@hiranya911 thanks for the clarifications. I'll work on it based on the Firestore listen API.

[ctrysbita]

@johndelavega Any news for real time database listeners?

[hiranya911]

I don't think anybody's working on this at the moment. But if somebody in the open source community wants to give it a try, they are welcome to. That's essentially how the Python Admin SDK got its listener API.

[johndelavega]

@ctrysbita I just did a pull request against the dev branch, you can check it out.

@hiranya911 Let me know if you're able to get it running.

[thatreguy]

Any updates on this ?

[johndelavega]

@thatreguy, thanks for the review.

The code is tested on Ubuntu cloud VM and it's been running continuously for over 2 weeks. A second test is running on Raspberry Pi Zero W (built as GOARCH=arm GOARM=5) and it's been running continuously for over a week. The test is using a particular auth flow using type: "authorized_user" json refresh_token credential file, like this: refresh_token.json

This tool create-refresh-token can be used to create the JSON file.

@hiranya911, Also tested it with "service_account" credentials and runs as great. I think this is a good starting point with basic functionality. Just like Python SDK, we can label it as experimental.

[rushib1]

will this feature ever be merged? I tried cloning the repo but getting use of internal package firebase.google.com/go/internal not allowed.

[johndelavega]

@rushib1, here's quick download then build, using go.mod (Linux or Windows):

git clone https://github.com/johndelavega/firebase-admin-go
cd firebase-admin-go
git checkout jd-event-listener
echo module firebase.google.com/go > go.mod
go mod tidy
cd cmd/listen2
go build

A binary executable should be created Edit cmd/listen2/main.go for firebase credentials and db name for your specific settings Rebuild, and real-time event listener should be working at this point

Run app with any command-line argument to initialize the db test path which is /user1/path1 Run a listener by running without command-line argument From a separate command-line console, run app with an argument to trigger an event The other console running the listener should show the event If the value is the same as previous, event will not trigger

Another way of getting it working: If you have existing setup either with GOPATH or go.mod Just copy these 5 files:

db/event.go
db/event_ref.go
db/event_test.go
internal/http_client_event.go
cmd/listen2/main.go

These source code are disjoint. Tested to work on the latest firebase-admin-go version 3.9.0 Tested disconnecting wifi on listener computer, then re-connecting, Listening continued OK.

Versions used for testing: go version go1.12 windows/amd64 and go version go1.12.7 linux/amd64

[rushib1]

thanks @johndelavega. I got it to work by replacing the code firebase-admin-go with your code and it works. I think this is a bad way to do it but was super easy to do.

[eliezedeck]

@hiranya911 any chance we'll get this merged anytime soon?

[hiranya911]

I'm somewhat hesitant to add this feature to the SDK. Adding this to the Python SDK resulted in a lot of confusion and issues (mostly developers not fully understanding the limitations of receiving realtime updated over REST). Is there any possibility of maintaining this feature as a separate extension?

[eliezedeck]

Wouldn't that be solved by explicit documentations? Javascript already has full RTDB integration, Go has lacked this feature for too long.

But concretely @hiranya911, what kind of confusion and issues? Any links?

[hiranya911]

Wouldn't that be solved by explicit documentations? Javascript already has full RTDB integration, .

The Admin Node.js SDK doesn't actually own the RTDB code. It is owned and developed as part of the JS client SDK. It gets included in the Admin SDK as a dependency. Furthermore, JS support for RTDB uses WebSockets. That is actually a better and more scalable way to support realtime listeners.

Go has lacked this feature for too long

When the RTDB API for Go was proposed it was understood that it will only ever be a thin wrapper around the REST API, and it will only support the simple transactions. Implementing listeners on top of REST is messy for several reasons -- one connection+goroutine per listener, periodic disconnects due to token refresh, no caching at the receiver end etc. These limitations actually make the implementation unsuitable for listening to nodes with a lot of data (too slow and too high download costs).

But concretely @hiranya911, what kind of confusion and issues? Any links?

Here are some from the Python SDK:

https://github.com/firebase/firebase-admin-python/issues/287 https://github.com/firebase/firebase-admin-python/issues/275 https://github.com/firebase/firebase-admin-python/issues/267 https://github.com/firebase/firebase-admin-python/issues/224 https://github.com/firebase/firebase-admin-python/issues/198

Our support team has also received quite a few similar issues. Overall, supporting this operation in Python has not been a pleasant experience. And given how few users actually want it, I'm partial towards not implementing it in Go, and pointing those who ask for it towards alternatives. I think Cloud Functions (which now supports Go) is a far better approach for implementing realtime listener scenarios at server-side. And if somebody really wants to run their own listener infrastructure, I'd point them to our Node.js or Java SDKs.

In any case, I'd like to talk to a few more people about this (specifically the RTDB team). Please keep the discussion alive by posting your ideas too.

[eliezedeck]

Oh I see, I didn't know that. Is there a particular reason the Websocket implementation has not been ported to Golang then?

[hiranya911]

Websocket protocol for RTDB is somewhat complex to implement (and even more difficult to get right). For Node.js and Java we essentially "borrowed" the already existing implementations from our JS and Android SDKs, which were mature and battle-tested. For other languages, we will essentially have to write it from the scratch. But there's not nearly enough demand for this feature in Go to justify committing our resources.

[smoya]

Any update on this?

[architkithania]

Would love to get an update on this. If it is not merged, could someone please provide me with steps on how I can actually make use of this feature if it is not in the main branch? Thank you very much

[eliezedeck]

By the looks of things, I don't think the team will implement this.

What I did, and might be useful to your @architkithania: I used the NodeJS admin sdk and couple it with go thru gRPC. Both Javascript and Go have excellent gRPC support. It works flawlessly so far.

[architkithania]

@eliezedeck Thank you so much for your response. I am quite new in cloud technologies and hadn't even heard of RPC. gRPC seems to be quite promising and may be just the thing I am looking for. thanks for the reference :D

[hiranya911]

I'm going to close this as we don't have any plans to support this API right now. If you think this feature is critical, please file an issue at https://firebase.google.com/support/troubleshooter/report and with enough feedback we can take another look.

[iamkhalidbashir]

dont know why this got closed as it is very important