Store Signals Locally if no Internet Connection

winsmith commented 3 years ago

This has been floated by a few people asking. Can the client store signals if the internet connection is not good enough. This has some implications:

Needs local storage

With this, we'd need to save signals to local storage (Defaults, a temp file, etc). This might be easy, but it might also be a problem if an app sends a lot of signals and fills up storage. Writes and encodes also might use up more CPU than just firing and forgetting the signal. Finally, the client should be able to run on all platforms that support Swift (iOS, macOS, watchOS, tvOS, Linux), and not all storage options might be available on all platforms.

Signals no longer transmitted live

This might need changes on the server: The time the signal arrives at the server is no longer necessarily the time the signal was generated. On the plus side, it might be possible to bundle up signals. Apps that send a lot of signals might use less energy that way, because they don't connect to the net for every signal.

Privacy Implications

I can't see no serious privacy implications in storing the locally generated signals on the user's device.

nik6018 commented 3 years ago

Storage Choices:

If we had a day's worth Signals using Defaults would load the entire thing into the memory before it would be transmitted to the server, so I don't know if this trade off can be made
Core Data would bring lots of goodies for storage management but it too has it's own can of worms (mainly migrations)
Haven't worked that much on storing data directly to disk, so can't comment on that.

Live Transmission

Uploading signals in bunch will truly optimise the process, this is big up side
We can have Live signals for Debug Mode only and then bundle up signals while in Productions (Firebase does this ... stream view can only be enabled with a flag in Debug builds and disable them in Production) (Sorry to repeat if you already knew)
I don't know if it's your product goal to display Live Signals but IMO they should be restricted to Debugging only (But I like them for my app though 😄)

winsmith commented 3 years ago

Why do you want to restrict live signals to debug only? I think live signals is a super neat thing, and I'd really like to keep them at least live-ish (I wouldn't mind a bit of a lag)

nik6018 commented 3 years ago

Well I thought it's a load thing on the server .... (Firebase tutorials given this reason for not enabling their continuous stream even view for production). But if we can have that without degrading server performances than it's a win :)

winsmith commented 3 years ago

So far we can. Let's see how the open beta changes that 😂

nik6018 commented 3 years ago

Cool. So were you able to give any thought on the storage part ?

winsmith commented 3 years ago

Probably needs to be a JSON file written to a temp location. I can't see any other way to make that work on all supported platforms easily.

nik6018 commented 3 years ago

Cool. Can I help in anyway possible ?

winsmith commented 3 years ago

I'm open to pull requests, if you feel like tackling this. Otherwise, you can help by finding out by thinking about when and how often the client should try and send data to the server, and without sending data multiple times, and posting your thoughts into this ticket. (This is an open invitation to all users of the client of course)

nik6018 commented 3 years ago

I think we can use a session based approach, where we gather all the signals for the current session and then when applicationDidEnterBackground(_:) gets called, upload all the signals using a BackgroundTask (we'll have at least 5 - 10 seconds to complete the operation once the app enters background)

winsmith commented 3 years ago

Grouping by session is a good idea. We shouldn't forget that the device might have no internet connection at backgrounding time though. Also, I'd like to send signals more often than just when the device backgrounds, so a periodic periodic uploader should be a good addition here.

nik6018 commented 3 years ago

We can implement something that checks for signal count. Uploading 10 signals in a bunch seems like a good conservative approach at least for the time being. So the utility will save 10 signals and once we have more than that we upload the last 10 and save the new signals.

winsmith commented 3 years ago

I like that!

nik6018 commented 3 years ago

Nice I can come up with a solution for this, for time being I'll start with Defaults as storage solution .... this will reduce the storage complexity initially

winsmith commented 3 years ago

Please do not use Defaults as storage. Using defaults as storage will cause the client to not work under Linux and Windows any more, which are supported platforms.

nik6018 commented 3 years ago

Aaah this is downfall of me coding for Apple Platforms only .... can you suggest any other method ? (Saving to file is the best way to go ?)

winsmith commented 3 years ago

I'd like this to be in a temporary JSON file. How about I get started with the general structure and then you can help out with PRs later?

nik6018 commented 3 years ago

Yeah cool no issues :)

On 16-Apr-2021, at 4:46 PM, Daniel Jilg @.***> wrote:

I'd like this to be in a temporary JSON file. How about I get started with the general structure and then you can help out with PRs later?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AppTelemetry/SwiftClient/issues/19#issuecomment-821104377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3HBNS5TJVNTGXECXVUP4DTJAMARANCNFSM42OMR2UA.

outcoldman commented 3 years ago

Also, consider using JSON streaming format (https://en.wikipedia.org/wiki/JSON_streaming).

This is how I would see the implementation.

The client should collect and store all signals on the disk, appending JSONs line by line to a temporary file to a known unique location of the app. In case of Apple apps could be

try FileManager.default.url(
    for: .libraryDirectory,
    in: .userDomainMask,
    appropriateFor: nil,
    create: true
)
.appendingPathComponent(Bundle.main.bundleIdentifier!)
.appendingPathComponent("apptelemetry.noindex")
.appendingPathComponent("data.json")

As you can see I store it in the library folder, so if the app will crash or Mac will reboot, we still have access to the old telemetry.

There are should be a background Timer (like fired once an hour) that will check on the disk if there is anything to send, will pick up those files and send them to your Telemetry server.

Preferable to send with the JSON streaming format, so there will be just one large HTTP call, and not bunch of small ones. Also, when sending, please consider using gzip-compression (to be honest I am not sure if URLSession does it automatically).

winsmith commented 3 years ago

JSON streaming looks super cool, I haven't seen that before. Thanks for that. GZIP should be enabled by default but it can't hurt to check that.

winsmith commented 2 years ago

This is now done, and I completely forgot to close the ticket. Oops.

TelemetryDeck / SwiftSDK