search

o19s / ubi

User Behavior Insights standard schema
Apache License 2.0
14 stars 4 forks source link

RFC/PROPOSAL: add timestamp to query.request #29

Open epugh opened 2 hours ago

epugh commented 2 hours ago

What/Why

What are you proposing?

We propose that the we add to the query request schema a timestamp field so that consumers of UBI data can know when the query was made. This timestamp field would mirror the existing timestamp field defined in the event schema:

https://o19s.github.io/ubi/docs/html/1.1.0/event.schema.html#timestamp

What users have asked for this feature?

We actually have in the OpenSearch UBI plugin that we track queries by a Unix Epoch field, so we capture the specific date time. But that is a different format than what we use for events. We also support a "date" but not a "datetime", so this needs fixing in the UBI plugin for OpenSearch anyway.

What problems are you trying to solve?

Consistency in the UBI specification and the UBI plugin, and moving to using ISO 8601 formatted time stamps everywhere to simplify down stream users of UBI data.

This bug was discovered by @alexeyrodriguez during his work to simulate UBI formatted data. Thanks @alexeyrodriguez!

Are there any breaking changes to the Spec

Yes, if you used Chorus for OpenSearch edition, then you would need to update. In the future this might cause a major version bump in the spec, but today, I think it's niche enough we use it as a chance to bump the minor version.

What is the user experience going to be?

Simpler consumption of the data.

Are there breaking changes to the User Experience?

Yes, if you used epoch formatted dates you need to switch to ISO 8601 format.

Why should it be built? Any reason not to?

Definitly let's change now before UBI becomes even more widespread ;-)/

What will it take to execute?

  1. Update UBI Spec
  2. Update UBI for OpenSearch Plugin schema defintion
  3. Update Chorus for OPenSearch to change format of dates being sent.
  4. Documentation and samples updates.

Any remaining open questions?

No.

epugh commented 1 hour ago

This is what the spec will look like.. Ignore the 1.1 version, this will be in 1.2...

image
jzonthemtn commented 1 hour ago

There is already a timestamp field in the queries mapping. What we can do is if the user provides a timestamp in the search request, the UBI plugin can use it instead of now.

epugh commented 1 hour ago

This is a revamped version, where we take out the required nature, and articulate two use cases for why you might want to provide the timestamp instead of letting the UBI plugin for the search engine do it.

image
jzonthemtn commented 38 minutes ago

Looks good. Linking https://github.com/opensearch-project/user-behavior-insights/issues/39 for visibility.