Use real time from an NTP server

[LikeTheSalad]

Android devices' clocks can be inaccurate, which can cause issues when analyzing distributed traces. One option that helps and doesn't rely on vendor-specific capabilities would be to get the real-time from an NTP server and set it in a custom Clock implementation when initializing the Android RUM agent.

[breedx-splk]

Are you aware of or does Elastic have prior work around this feature? I think this seems like a good idea, perhaps with a couple small downsides:

• increased network overhead (tho ntp is pretty small indeed)
• slightly more complicated clock behavior when ntp is unavailable or when telemetry is created during ntp time sync.
• time-based correlation between otel data and in-app data sourced from system clock becomes difficult.

[LikeTheSalad]

We have it in Elastic atm, I'd say the most concerning downside is when some telemetry is created before the ntp clock is available, although the TrueTime lib caches the time in the disk so the issue should only happen the first time the app is open (unless there's a network connectivity issue or something that prevents the request from succeeding). I'm also not sure if we should do some periodic updates to it even if it's already cached, so I think there's quite a lot to discuss about it in general, however, it seems to be the only solution that only depends on the mobile device, there might be other alternatives to this issue but would require making the servers aware of it afaik. I think @marandaneto has some insights on this.

[marandaneto]

Yeah, I think the TrueTime is the most reliable one - on device, other alternatives depend on clock drifting on the server which we cannot control as a vendor-agnostic lib. I shared a bit of context about this topic here.

[marandaneto]

Might be needed for v1, assuming a device has the wrong date and time, and after upgrading the SDK with an NTP server, the telemetry timestamps may seen off (before and after the upgrade).

[LikeTheSalad]

Update: It seems like Android might have come up with a [solution](https://developer.android.com/reference/android/os/SystemClock#currentNetworkTimeClock()) (for API 33+ though 😅).