Fetch and refresh matching contexts in the background

[JalilArfaoui]

Currently, we only fetch matching contexts when an URL is captured, and each time it’s captured. This arises multiple issues :

• Unnecessary traffic (Up to 1 MB, getting the whole list of all subscribed matching contexts) and battery consumption
• Query can sometimes be long to respond, and depends on mobile network speed, thus creating a delay between navigation and notice display
• Creates unnecessary load on backend infrastructure

Let’s put that in a background service, as we have on https://github.com/dis-moi/extension …

[felix-lambert]

To store (cache) the matching-contexts in the Android app, I’d go either with App preferences, or Database, not sure yet

https://stackoverflow.com/questions/23342603/android-do-database-operations-consume-power https://www.youtube.com/watch?v=LHaxq3YdxZU

I think we will run with the same problem as now. We write to the database each time we follow or unfollow someone. We write every 30 minutes. Also we read it each time we call an url + building a new service working in background. IMHO this scenario will not significantly improve the battery life and add a big amount of complexity.

So, again, if you think that I'm raising a valid problem, I would propose to do a meeting (talking about the mobile, the backend...) and see how we can simplify the diagram of https://github.com/dis-moi/backend/pull/287 (even if it's possible). See that there is maybe a more global problem.

If it's not valid, go for it :)

[JalilArfaoui]

I’d be curious if you could just throw some idea here (even incomplete and approximative) about how we could simplify the global flow. I would be more happy to think and talk about it !

Now, about power consumption, I can see your link where some developer want to write to the database every second. It’s far from what I’m suggesting. Also, maybe storing matching contexts in shared preferences, or in memory or in some kind of cache memory would be better, I’m not sure about that.

What I know is that on the extension side we manage to maintain the whole list in memory, with no noticeable performance drawback, so I think a complete Android app can do at least the same.

And I can’t see how fetching the matching contexts list on each URL navigation can be a suitable long-term solution. And I can’t see how reading the same amount of data locally instead of remotely can be slower or more consuming …

So, again, please suggest a better and more simple solution.

Also, about the backend, again, PR welcome !

[JalilArfaoui]

From Official guide to app architecture

I’ve done the retrofit part. I’m now building the repository and model.

[JalilArfaoui]

Persistence is ideal for the following reasons:

• Your users don't lose data if the Android OS destroys your app to free up resources.
• Your app continues to work in cases when a network connection is flaky or not available.

By basing your app on model classes with the well-defined responsibility of managing the data, your app is more testable and consistent.

[felix-lambert]

I’d be curious if you could just throw some idea here (even incomplete and approximative) about how we could simplify the global flow. I would be more happy to think and talk about it !

My solution would be to use tokens and encryptions by creating a vault key and to make it more secure why not use PBKDF2.

Now, about power consumption, I can see your link where some developers want to write to the database every second. It’s far from what I’m suggesting. Also, maybe storing matching contexts in shared preferences, or in memory or in some kind of cache memory would be better, I’m not sure about that.

What I'm trying to say is that you have to call your sqlite database or your shared preference each time you call a new url with all the matching contexts. These services are not made for what you're trying to do: Shared preferences allow you to store small amounts of primitive data. The sqlite database is for helping the user still having data when he's offline. For example by using firebase and react native, the firebase library handles my data by using sqlite, but this data is just the data that the client needs. The matching context will get 100, 10000, 1000000 times bigger with time and it's not the role of the client to store on a phone every 30 minutes such a big amount of data: it will take some space to the client's phone and it will consume unnecessary battery consumption.

What I know is that on the extension side we manage to maintain the whole list in memory, with no noticeable performance drawback, so I think a complete Android app can do at least the same.

LocalStorage is commonly used to keep data that would be convenient for the user to see even if the browser was refreshed. For example, many forms save a user's input in LocalStorage until it is submitted, or user preferences, like a UI theme. Or it's also used to keep the user online. However, LocalStorage should not be used for large amounts of data like a big list of matching contexts. Maybe for now the list is at an acceptable size for the browser. But with time it will have performance drawbacks.

And I can’t see how fetching the matching contexts list on each URL navigation can be a suitable long-term solution.

I never said it was a long-term solution. I thought that with the new api version we would solve the problem of: getting all the list of matching contexts on the client side and save it here. So when I would be able to get a reasonable amount of matching context from the server, yes I cannot see a simpler solution than dealing with it directly.

Also, about the backend, again, PR welcome !

I'm actually thinking of doing a node server that handles data with tokens so that I can give encrypted data to the client, but also exactly what the client needs.

[JalilArfaoui]

My solution would be to use tokens and encryptions by creating a vault key and to make it more secure why not use PBKDF2.

Interesting ! Let me know when you have some details (how generates the token ? what does it contain ? where does the matching between url and pattern take place ? …)

The matching context will get 100, 10000, 1000000 times bigger with time and it's not the role of the client to store on a phone every 30 minutes such a big amount of data

We’re fetching them all anyway currently, since the matching occurs client side … so let’s at least fetch them least often … We do have some leads about how to reduce the amount (we’re already fetching only followed contributors, we’ve talked about incremental fetching, and bloom filter … but I’m curious about new ideas that you may have

exactly what the client needs.

which is ?

[MaartenLMEM]

Hi there,

To help some functionnal remarks :

This discussion tackles the specific issue we have to deal with our contextual service : recognizing a situation to send contextual information. This is a big difference with many apps.

The only way to start recognizing a situation is having sign of recognition / token of recognition of the situation, that we name matching context. Concretly, it is firstly the page url, and, secondly, if an url matches, the matching context engine may look in the page other signs with xpath.

That's the job the matching engine has to do.

That s why the app needs to keep in memory a minimum amount of _signs of recognition - currently a list of url matching contexts. - and, for each matching context, an object : currently an notice number that is related to a message that will be displayed to the user, because it matches to his situation.

And currently we have a performance issue on this engine on the app.

[felix-lambert]

Interesting ! Let me know when you have some details (how generates the token ? what does it contain ? where does the matching between url and pattern take place ? …)

I'm aware it's quite tricky but maybe we could think of doing something getting close to this?

Something to note: I'm sending the contributor id + the crypted domain name to get the matching contexts without domain names in 4.

I think it can be improved. But the best is to have a meeting so that we could know what is acceptable/unacceptable in term of privacy... I need to have some info so that we can propose something that can get even closer to our needs.

[MaartenLMEM]

Please tell me if I go to far on technical part but I have some comments and questions based a previous one of Felix :

I think we will run with the same problem as now. We write to the database each time we follow or unfollow someone. We write every 30 minutes. Also we read it each time we call an url + building a new service working in background. IMHO this scenario will not significantly improve the battery life and add a big amount of complexity.

Beforehand, I do not think we will run with same problem. I d rather say that we avoid current issue described by Jalil formerly, but we may generate new issues, that seems to me less important (since for example it won't generate delay to fetch and display a notice), but that are nevertheless to take seriously.

We write to the database each time we follow or unfollow someone.

This action is rare. I think it has no impact.

We write every 30 minutes

Good point. I see two options here if we really need a less battery consuming solution here.

1. Refreshing matching context :
   • If Dismoi app is opened, each 5 minutes for example
   • If Chrome is launched, each day.
2. Other option has been is optimizing the refresh process of notices and matching context by telling what s new since last time the app connected (Jalil came with this idea some months ago)

Also we read it each time we call an url

It seems me to be the main job of DisMoi that we cannot avoid. How we optimize this reading act is the question here.

• building a new service working in background.

I will need some insight here. Because I thought service working in background here is just refreshing each 30 minutes and others actions that we have to run in any case.