generalise the dashboard to work with the e-mission-phone master branch

asiripanich commented 4 years ago

The dashboard still has many components, mainly those that responsible for querying data, specific to my own e-mission-phone branches (rciti1 to rciti4).

To generalise this it would involve knowing the database structure of the master phone branch.

shankari commented 4 years ago

@asiripanich I might be able to help with this as part of the upcoming Agile Urban Planning deployment. The dashboard looks very cool and I think it will really help the deployers work closely with the pilot participants. They have a fairly tight deadline, so no guarantees. I will post back here when I start work on it so we don't duplicate our efforts...

asiripanich commented 4 years ago

Thanks! that would be great. Just to let you know I'm no pro in Shiny, so feel free to suggest changes to the core structure. By the way, I'm using the golem framework for this project.

shankari commented 3 years ago

Hi @asiripanich, we are finally starting to take a look at this (@shankari and @ahcyip). It looks like we mainly need to change the queries.

According to the README, the queries are in R/utils_stage_db_queries.R but that file is currently empty. Can you let us know where the queries are so we can estimate the effort required for the change?

shankari commented 3 years ago

By poking around the code a bit, it seems like this is maybe now in R/fct_stage_db_queries.R. Is that correct?

asiripanich commented 3 years ago

@shankari sorry you are right, the query functions are in R/fct_stage_df_queries.R.

asiripanich commented 3 years ago

I'm happy to help if you can provide a toy database of your study. ;)

shankari commented 3 years ago

I can generate a database from my data 👍

But thinking of the high level before that, it seems to me that we can separate the visualization into the common and customized parts. The location, motion_activity, trip, section etc objects are likely to be common. The manual/*_confirm objects are the ones that are likely to be customized.

Would it make sense to pull out the code that deals with manual/*_confirm objects into a separate module? Then we could have different versions of that which deal with survey_confirm, mode_confirm, etc. And when people installed the dashboard, they would install the base emdash and then the confirm module related to their study.

They may have to write this module, but it should be straightforward.

@asiripanich @ahcyip any thoughts?

asiripanich commented 3 years ago

@shankari I totally agree with the idea to generalise the codebase the way you are proposing.

shankari commented 3 years ago

@ahcyip, the tasks are then:

fix the CI so we start with a clean slate. This also allows us to become more familiar with the code
refactor the code by moving the following survey specific code into their own module:
- query_trip_ends from R/utils_stage_db_queries.R
- tidy_trip_ends from R/utils_tidy_data.R
- query_trip_ends(cons) %>% tidy_trip_ends(.), from R/mod_load_data.R
design choice: you can either make a single handle_user_input module that both loads the data and tidies it, or you can have two separate modules to mirror the original structure
ensure that everything works without any user input; using only the automatically sensed trip information
ensure that everything still works even when the survey modules are included
write user input modules which read the mode_confirm and purpose_confirm objects from master instead of the survey_confirm modules here
ensure that everything still works when our modules are loaded instead of the survey modules

shankari commented 3 years ago

@asiripanich did I get everything?

FYI, we also need to get confirmation from NREL that we can contribute to this project. I don't think it should be a problem, but I haven't contributed to external projects since May, so I should double check.

asiripanich commented 3 years ago

Sounds good to me. Can you also send a standard e-mission database to be used for dev and unit testing?

shankari commented 3 years ago

standard e-mission database to be used for dev and unit testing

Sure. Should this have any user annotations? If so, in which format? And what format do you want it in? mongodump? JSON files?

shankari commented 3 years ago

FYI, I really want to get this set up quickly, and @ahcyip thinks that the refactoring is too complex, so I am going to hack around this by removing all references to the survey code and only displaying the trips.

We will try to do the beautiful version in November/December when I get a dedicated intern.

asiripanich commented 3 years ago

And what format do you want it in? mongodump? JSON files?

With user annotation in mongodump format would be great.

FYI, I really want to get this set up quickly, and @ahcyip thinks that the refactoring is too complex, so I am going to hack around this by removing all references to the survey code and only displaying the trips.

We will try to do the beautiful version in November/December when I get a dedicated intern.

No worries. :)

shankari commented 3 years ago

I started making these changes because I hadn't written code for a week and I was getting antsy.

@asiripanich I can submit my preliminary changes as a PR to a separate branch if you create one. I am currently at the error below and I don't know how it ever worked, because the trip objects only have user_id not uuid.

We can discuss further in the PR :)

About to load trips
 [1] "user_id"                          "metadata.key"
...
Warning: Error in normalise_uuid: `uuid` is not a valid column name in `.data`.
  [No stack trace available]

asiripanich commented 3 years ago

:) Haha and you choose to write some R code to help with that. I often hear people with software engineering background say that they hate R because it is a weird language.

Sure I'm happy to review your PR. Don't forget to send me your test data so that I can test.

https://github.com/asiripanich/emdash/tree/vanilla

shankari commented 3 years ago

well, R is a weird language. I think it is because most of the languages that software engineers are familiar have syntax that is C-like. So if you have written Java code, for example, Objective C, Scala, Javascript, Kotlin etc are not that different.

R has also these weird $>$ and <- syntax and stuff with . I've been guessing what everything does and it is much harder to guess in R.

ahcyip commented 3 years ago

R pipe %>% and dot .

x %>% f(y) %>% g(z) g(f(x, y), z)

x %>% f(w, .) f(w, x)

asiripanich commented 3 years ago

Implemented in #4.

asiripanich / emdash

generalise the dashboard to work with the e-mission-phone master branch #2