Ability to purge old logs

[mstenta]

This came up on the monthly call. Over time, Field Kit can become littered with lots of old logs. Perhaps we need something like a button that lets you purge logs that have been synced.

[mstenta]

Cross referencing this to the discussions in #282 and #284

[jgaehring]

I believe there are some ideas in #243 as well that are relevant, regarding how to manage logs that multiple modules may have access to.

[jgaehring]

Perhaps we need something like a button that lets you purge logs that have been synced.

I'm torn between this option and providing a way to select multiple logs at a time and perform bulk deletions. The latter option would be a little more complicated, because we would need to provide some considerable UI elements to support it (with the ability to "long press", ideally), but I feel like it would provide coverage for more use cases, and add more flexibility in general. It would also make possible other bulk operations, potentially, like the ability to mark multiple logs as "done" at one time.

[mstenta]

My 2 cents: I like the idea of a granular manual option like you describe. I think we may also (and maybe first) have an automatic option that purges old logs after a certain threshold. Left alone, the list will just continue to build up, and users probably won't go through the extra step of manually deleting most of the time. Perhaps this automatic option could be toggleable, so that you can take control and manage them manually if you want, otherwise it will delete logs after X days or something.

[jgaehring]

Hm, ok, I think we may be describing of two distinct use cases though:

1. The user no longer needs a log on their device, as determined by some criteria (not exactly sure what that would be), and no longer wants them cluttering the device UI.
2. The number of logs exceeds a certain threshold and should be purged to avoid storage and/or performance issues.

I believe the first is the original issue that came up in the Monthly call, and I still believe the best way to deal with this is to let the user decide, and just make it easier for them to execute on that decision by providing a bulk delete option. I think the decision tree for determining what logs "aren't necessary" any more is too complex to try to model at this point, and we wouldn't be able to predict the users' needs well enough to perform such an operation automatically.

I believe the second case would be best handled by making use of each module's log filters, and perhaps requiring that each module provide some sort of date filter, so periodically it can be determined by those filters which logs no longer need to be kept, and so can be deleted from IDB. The decision about what logs meet the criteria can then be handled on a module-by-module basis, where I think each module's limits and needs can be judged more effectively, rather than trying to establish some sort of global criteria for all modules.

In any event, more I think about it, I don't think either of these has a particularly simple solution, and each should probably be addressed independently based on user needs and potential problems as they arise.

Also, I think the whole situation would be ameliorated by the Log Manager module I've proposed in #243.

[mstenta]

Ah OK so I was referring to the first one (user no longer needs a log on their device), which I agree is what came up on the monthly call. If I remember correctly, it was also related to the fact that the sort order was not based on log timestamp, so it meant the user had to scroll all the way to the bottom of the log (which continually grows) to get to the latest log they just created. So #246 helped with this a bit.

OK what about this as a short-term compromise: a button that simply lets you "Delete All" logs. This would clear out the local storage on the device, and allow you to then click the "Sync" button again to refresh it.

So user story: I've been taking observations for a few weeks now, a dozen a day, I have close to a thousand logs. They're all synced to the server regularly, but I still have a copy on my phone. I know everything's synced up, I don't need to worry about selecting which logs I want to keep and which I don't. So I just click "Delete All", followed by "Sync" and I'm ready to keep going with a clean slate.

[mstenta]

The "Delete All" button could first check to make sure everything is synced and there are no outstanding changes, and show a warning or outright prevent you deleting everything, until after that is resolved.

[jgaehring]

Hm, yea, that might be a good stop-gap. Maybe we hide it in the "more" menu in the upper right corner of My Logs. Will also want to be emphatic that this will delete EVERYTHING.

The "Delete All" button could first check to make sure everything is synced and there are no outstanding changes, and show a warning or outright prevent you deleting everything, until after that is resolved.

I'm still a little stuck on the easiest way to implement that w/o risking data loss.

I guess, taking a step back, I wonder if a stop-gap is even needed at this point. Not sure I've heard from enough users at this point that it's worth putting redundant efforts into. Perhaps a better stop-gap would simply be resolving #306, since that needs to be addressed anyways, and would achieve more or less the same goal, but with less effort.

[mstenta]

I'm still a little stuck on the easiest way to implement that w/o risking data loss.

Maybe we don't need to have any complicated logic checks to start. Just show a big red warning saying that it will clear out all logs, synced or not, like we do on logout (it's no different from that). And hide it in the top menu, like you said. That will work as a first step.

306 is a workaround, but it is not a solution to this problem. Requiring users to log out and log back in is MUCH more time consuming than a three-click delete-all process. It's less effort for us to implement that (and it should be fixed regardless), but logging out and logging back in is not a fast process, so it's more effort for users.

Not sure I've heard from enough users at this point that it's worth putting redundant efforts into.

I've heard from at least 3, and as more use it I won't be surprised to hear from more.

306, since that needs to be addressed anyways, and would achieve more or less the same goal, but with less effort.

That needs to be fixed, I agree. But that is a workaround at best, and if someone is thinking to themselves "how do I clear out all these logs?" logging out is not going to be the first thing that comes to mind IMO.

Let's look at other apps, like Gmail, which also stores data locally. As a Gmail user, I don't have to think about whether or not my mail is synced, it happens automatically in the background. And it has a setting for how long into the past it should keep emails (I think the default is 30 days). I'd love to see that kind of automation in the future, but I understand we may be a long way from that. A "Delete All" button is a simple (I assume?) feature we can add now and remove later when it is no longer relevant.

[mstenta]

Let's see if we get more feedback on these ideas after the v1.0.0 release.

[jgaehring]

I'm still stuck on this issue. I feel like it's got a bunch of competing issues tangled up in it. We've got an overloaded feature set in My Logs (is it a ToDo app? is it an observation app?), some unclear distinctions of what "delete" actually means, an all-or-nothing caching system, and inflexible syncing mechanism.

I'd really like to punt on this until milestone 0.9.0, when we can focus on better defined use cases, where I think the path forward will be much clearer. But I'm not sure if this issue demands more urgency.

Part of me is optimistic this issue will just "go away" once we settle our syncing and caching issues. Particularly #367/#385. If we just apply the simple rule to caching that we only cache a log if it's 1) timestamped within the last 30 days, 2) timestamped within the next 15 days, or 3) has unsynced changes, and then periodically flush that cache, we should be in a pretty good state. Maybe that combined with a different way of organizing our logs as "Upcoming", "Late", and "Done", similar to the way the Spraying module does it and how the farmOS dashboard has always done it.

I'm also wondering if it might make sense to change the "delete" icon and terminology to "hide" or "archive" or something that indicates the change isn't permanent. I feel like at some point we will want to the ability to actually delete logs and we will have set a bad precedent for what it means to "delete" something.

[mstenta]

If we just apply the simple rule to caching that we only cache a log if it's 1) timestamped within the last 30 days, 2) timestamped within the next 15 days, or 3) has unsynced changes, and then periodically flush that cache, we should be in a pretty good state.

This sounds right to me - I think the main issue is that over time you end up with an ever growing list of logs. But if we "periodically flush that cache" then that would serve to purge the stuff that's no longer relevant (as defined by the cache filters you describe). +1

[jgaehring]

Cool! Yea, hopefully that's sufficient. Down the road, perhaps we can even make it configurable.

I was thinking more about this last night and thinking the Late/Upcoming/Done layout could really help, and it would make even more sense if we start steering the My Logs more towards being the dedicated ToDo module. I've gone back and forth on whether My Logs should be replaced by a dedicated ToDo module, or it should become the ToDo module itself. Still not sure about that, but either way, it seems expedient to have it cater towards the ToDo use case in the meantime.

[jgaehring]

I was thinking more about this last night and thinking the Late/Upcoming/Done layout could really help

See #399.

I think the goals of this issue have been achieved by the new syncing and caching API's, especially with 1796f8d.