botpress / messaging

Botpress messaging server
42 stars 37 forks source link

feat(events): support for "user fetched" event #542

Closed AbrahamLopez10 closed 1 year ago

AbrahamLopez10 commented 1 year ago

This will be used in the runtime to track "repeat users" of bots for Cloud Analytics.

linear[bot] commented 1 year ago
CLS-105 Analytics - Runtime: Track bot users, sessions, and messages

* Analytics events will be sent by the chatbot runtime over HTTPS to the Amazon API Gateway endpoint as a collection of one or more JSON objects that conform to the [CloudEvents specification](https://cloudevents.io/). * The event types sent by the runtime will be: * new_user * session_started * message_received * The event data sent by the runtime will be: * Bot ID * Timestamp * User GUID * User’s timezone * Channel (Whatsapp, Telegram, web chat, etc.) * Bot version (last modification date?) * The HTTPS request that delivers the events should be done in a fire-and-forget fashion in the runtime where the delivery of the analytics events is performed asynchronously and independently of the processing of messages by the runtime.

AbrahamLopez10 commented 1 year ago

Hey @samuelmasse, many thanks for the feedback! I've been trying to navigate my way through the messaging codebase as I'm not very familiar with it. Let me reply to each point you made so we can confirm whether we need to adjust the current approach or choose a different one:

It's completely arbitrary that this would work at the moment, considering it's out of scope for the fetch function to be a tool used to measure returning users. Someone might create a new feature at some point that fetches user information at random intervals that would not correlate with user activity and that would pollute the metric.

Yes, I realize the "user fetch" event could be fired off multiple times at some point, and was thinking of tracking messaging user IDs on the runtime side to actually track repeat users.

A returning user is someone who interacts with the system some significant amount of time after their last interaction. Something being fetched is not an interaction, someone refreshing their screen, loading a website and ignoring the chatbot box, etc these are not returning users.

We already have a separate metric for that called "sessions", that will be tracked purely on the runtime side.

So the way of implementing this would be to create a new service (maybe something like UserMetricsService), you give MessageService as a dependency to that service and you listen on new message events.

My intention was to keep the changes to messaging to a minimum and make them as abstract and generic as possible and not introduce any specific details of analytics.

Also this looks to me like a backend only feature, I don't see why the websocket should get notified when a user is fetched. The websocket has the ability to fetch the user info anyways, so why do we need to send it a confirmation that the user info was fetched on top of that?

If we leave it up to the consumer to use or ignore this "user fetched" event, what are your concerns about introducing it?

On the other hand, is there any way we could reliably track real "repeat users" (i.e. a person) in messaging, which are different from a "session" (a new interaction with the chatbot)?


That aside, I'm wondering if you could help me with the following quick questions:

  1. Is the user fetched once per conversation or every time a message is processed?

  2. Is the user ID the same across multiple conversations for all channels? (including web chat, assuming it uses cookies or something similar to remember the user)

Thanks!

AbrahamLopez10 commented 1 year ago

Cancelling PR for now as we'll follow a different approach to track new vs. repeat users for analytics.