Datapoint download overload issues

[valllllll2000]

When there are many form instances per datapoint, the backend is not able to process the request on time and the datapoint retrieval fails with a 500 Error. To fix the issue here is what will be done:

Create a new endpoint to serve smaller chunks of data to the App

• Get first only datapoint list (only the metadata). Datapoint list already has a cursor
• For each datapoint, retrieve assigned form instances. Add cursor to track form instances

Issues

• App deletes empty datapoints, remove that logic
• The images are downloaded after each datapoint batch. It is better to get them either:
  • at the end of the whole process
  • after each batch of survey instances

[valllllll2000]

With the new api changes, we will first download all the empty datapoints and then their corresponding survey instances. While the datapoints are being downloaded we had to add a new status: DOWNLOADING.

Here is how the datapoints look (we would need a new icon @Kiarii what do you think)

While those datapoints are being downloaded, we do not want the users to open them as we do not know yet which forms have been submitted. As you can see there is an error message SnackBar shown at the bottom with a message, what do you think @janagombitova

[janagombitova]

@valllllll2000 Sounds fine by me. Just to check a few things:

• Users will only be able to submit new monitoring forms, once all data points and their past submissions (for the assigned forms) are in?
• Will the Error snackbar show once Al clicks on a data point?
• Are we showing something indicating progress while the downloading is happening?

[Kiarii]

Not sure if this is open or not but some background questions and thoughts:

• How long does it take for the datapoints to be actionable - i.e. time between their appearing on the screen and being openable?
• Is the time precisely determinable or indeterminable?
• Is it possible to be informative? Either by time or number of datapoints to be downloaded?
• if it takes long, then we can hold off presenting the datapoints until they and their respective data are in; a number of possibilities can be considered. If datapoints appear on the screen, users assume they are actionable; presenting datapoints and making them inactionable (pending download) goes against that intuition
• if it not long, then we can present them as they are loaded but lock interactions on the screen with a loading component..

[janagombitova]

@iperdomo and @kardan can you clarify the questions Kiarri shared so we can make a decision on how to improve the UX/UI?

[iperdomo]

• How long does it take for the datapoints to be actionable - i.e. time between their appearing on the screen and being openable?

Unknown - it depends on network conditions, server load, device

Is the time precisely determinable or indeterminable?

Unfortunately we don't have anything in place to estimate the remaining downloading time.

Is it possible to be informative? Either by time or number of datapoints to be downloaded?

Right now we know the number of synchronized datapoints at the end of the process. There is no way to know upfront how many datapoints the device is going to get (due to technical limitations in the backend)

if it takes long, then we can hold off presenting the datapoints until they and their respective data are in; a number of possibilities can be considered. If datapoints appear on the screen, users assume they are actionable; presenting datapoints and making them inactionable (pending download) goes against that intuition

This is something we can do. The datapoints that are not yet "ready", have an DOWNLOADING status. We can filter out (hide) those from the list we present to the user.

if it not long, then we can present them as they are loaded but lock interactions on the screen with a loading component..

Again, the synchronization speed depends on several physical factors.

I vote for hiding those DOWNLOADING datapoints from the list until they're ready. @Kiarii agree?

[Kiarii]

yes, for now @iperdomo hiding downloading datapoints seems better for handling the situation..

However, we need to communicate the content (dps) loading process and when it has ended, otherwise, dps will appear as if by magic and for huge dps dumps Al/Salim will have no idea if download is still ongoing or done. Since we dont have the numbers to give a clear message we could use an indeterminate progress indicator (I am assuming if the process timesout we have notifications to handle such errors). In this case we can hold off showing dps until they are in.. assuming that this task of setting up devices takes place in the office/cafe/.. and not during an in-field datapoint monitoring exercise..

Later, if we deem that the process will be painfully slow, then we can use notifications to notify Al/Salim when an assignment download begins, and ends. The assignment incl. all respective data in this case are presented after successful download.

[janagombitova]

Are this issue and the changes still relevant considering we took a different approach to fix the data point sync issue?