Read only data for the past week by default

[shankari]

Again, based on our experience from emdash, the default should not be to read data for a short duration (ideally one week/one month). If users choose to read more, they are welcome to.

Right now, the default is everything, and we read it as soon as the user has logged in to the dashboard. So we literally perform the most resource intensive query immediately. When I tried this on one of the admin dashboards, both the user and the trip tables were unresponsive, and I found these errors in the console.

The webapp layer was essentially unresponsive.

I think that we discussed this in one of the early meetings related to the dashboard as well.

[shankari]

we might also want to change all the implementations from find_entries to get_data_df as well since we only read one key at a time anyway. I don't think it will necessarily be a performance improvement but it might be with a different implementation!

[shankari]

• we currently support filtering after the initial data load, but that is a problem because you cannot even see the filter range until the initial load complete
  • this is pretty terrible from a UX perspective, because the dashboard is essentially unresponsive until the initial load is complete, which can take minutes on staging and so on
• you should never start with the computationally intensive operation and then filter down to the smaller dataset for an interactive application
  • instead, you should start with the smallest meaningful operation so you can show something immediately and then if the user chooses, allow them to start a more computationally intensive operation

which is why, I suggest reading for the past 7 days on login

Note that in this case, the initial screen that we see when we login cannot show the full sign up trend and trip trend by default because we will only load one week of trip data initially.

So we may want to think about UX design of that screen. Should we show the last week and clearly label it as "last week data" or should we just remove the trip trend and show something else meaningful instead?

I am fine with starting with something reasonable (e.g. last week trip trend) so we can get this out and then redesigning later.

Note also that not all of our tables and data are time dependent. So the participant-level data such as the user table and the demographic table need not be filtered. But the trip table and the trajectory table should be filtered by default.

I guess an alternate option might be to also read the trip table only when we access the tab. But I think that is complementary to the one week issue. Because we still may have 20,000 trips and even when we access the trip table, we don't necessarily always want to load it all and hang.

So that is a separate change, but we might as well do that when handling this scalability change.

[achasmita]

• we currently support filtering after the initial data load, but that is a problem because you cannot even see the filter range until the initial load complete

  • this is pretty terrible from a UX perspective, because the dashboard is essentially unresponsive until the initial load is complete, which can take minutes on staging and so on

• you should never start with the computationally intensive operation and then filter down to the smaller dataset for an interactive application

  • instead, you should start with the smallest meaningful operation so you can show something immediately and then if the user chooses, allow them to start a more computationally intensive operation

which is why, I suggest reading for the past 7 days on login

Note that in this case, the initial screen that we see when we login cannot show the full sign up trend and trip trend by default because we will only load one week of trip data initially.

So we may want to think about UX design of that screen. Should we show the last week and clearly label it as "last week data" or should we just remove the trip trend and show something else meaningful instead?

I am fine with starting with something reasonable (e.g. last week trip trend) so we can get this out and then redesigning later.

Note also that not all of our tables and data are time dependent. So the participant-level data such as the user table and the demographic table need not be filtered. But the trip table and the trajectory table should be filtered by default.

I guess an alternate option might be to also read the trip table only when we access the tab. But I think that is complementary to the one week issue. Because we still may have 20,000 trips and even when we access the trip table, we don't necessarily always want to load it all and hang.

So that is a separate change, but we might as well do that when handling this scalability change.

I am currently working on displaying a week data in the initial load screen for trip table and trajectories table. I have few questions:

• Trajectory table loads only after we select respective tab so :

  • One option is to leave it simailar way and load a week data when user select the tab initially.
  • Another option is to make it load similar to other tab(we were loading it separately as we have huge volume of trajectory data, but now we are loading only a week data at first, both ways will work.)