Open sync-by-unito[bot] opened 1 year ago
➤ On 2022-10-28, Nicola Cabiddu commented:
[~jonathan.reams@mongodb.com] assigning this to you.
➤ On 2022-10-28, Jonathan Reams commented:
By the time a timestamp is being stored in a subscription, the query that created the timestamp has already been created with whatever precision it has. I think the server should just choose a rounding mode to the next millisecond and use that. If we do the rounding so the server sends more data than the exact local query would match then the user will be able to query more precisely on it locally anyways.
➤ On 2022-10-28, Michael O'Brien commented:
So, the server currently does choose a rounding mode depending on the operator in cases where it can do so in a way that guarantees that the result set will not diverge from the client's view.
Here's a summary of the current server behavior:
So, (1) and (2) already guarantee correct results without the user needing to modify their query in any way for many of the most common use cases. Number (3) is the case where we could try to improve the behavior by intercepting this on the client - either throw an error if the timestamp in the query has sub-millisecond precision (essentially, ask the user to tweak the query so that the server can handle it under case (1) above) or do the rounding for them.
Addressing (3) by doing server-side rounding seems like it could introduce odd behavior - because it means the server would be returning a result set that wouldn't actually match what a client would get if it evaluated the same query locally. That might have implications for things like client reset recovery, but in general seems like something we should be careful to avoid.
➤ On 2022-10-28, Jonathan Reams commented:
[~ian.ward@mongodb.com], should we just not have nanosecond-precision timestamps in realm? Or should you just not be able to query on them?
➤ On 2022-10-28, Michael O'Brien commented:
There's another possible quirky behavior related to this where a client might see a different value for a timestamp field depending on whether it received it from a bootstrap (i.e. pulled from the state store in mongo, in which its represented as a Date object with millisecond precision) or a history entry routed to the client during steady-state (a realm timestamp value that hasn't been roundtripped through the mongo Date representation, so it still has full nanosecond precision).
I think in an ideal world, the realm timestamp data type would have the same precision as mongo's; a user who wants to store nanosecond precision timestamps could always still represent them as an integer, if that's desired.
➤ On 2022-10-31, Ian Ward commented:
Yeah I don't think we change MongoDB's server storage right now - so we should just align with MongoDB. Would this be a breaking change? I thought this was already settled though. Don't we just drop the nanosecond precision or something?
➤ On 2022-10-31, Jonathan Reams commented:
Yeah, the problem is that you can construct a Query
at nanosecond resolution which won't match anything on the server. I think we can make it so that if Sync is enabled the client truncates precision to milliseconds.
➤ jedelbo commented:
If we change realm to only support millisecond resolution, then I guess we can store the timestamp in a 64bit integer only. That would simplify a lot of things, but would of course be a breaking change.
➤ Jonathan Reams commented:
Added discuss label so we can find a path forward here. Fixing this so realm timestamps match the precision of mongo timestamps is a big task, but we should do something here so that querying on timestamps works correctly. Maybe we can roll this into next-major?
➤ Michael O'Brien commented:
I like the idea of keeping the data format the same, but just truncating it to millisecond precision. This might still qualify as a breaking change for certain behaviors? but at least its equivalent from a file format POV
There is an oddity at play in that MongoDB has millisecond precision and Realm uses nanosecond precision. This means that when we receive a query from Realm on a timestamp that has non-zero nanoseconds we can't safely convert that into a MongoDB date time query in such a way that we could guarantee the same results we send is what a local realm query would return.
I am not sure what the best route forward is, but I wanted to file a ticket to have it exist here in the hopes we can align on this at some point. This could include:
Error we generate:
Relevant Link: https://github.com/10gen/baas/blob/b5d4ba5013c044a9da4fc4358ecd894d0cc50b6e/realm/rql/mql_convert.go#L502