search

e-mission / em-public-dashboard

A simple and stupid public dashboard prototype.
BSD 3-Clause "New" or "Revised" License
1 stars 23 forks source link

Adding Survey Responses to Public Dashboard #124

Closed Abby-Wheelis closed 4 months ago

Abby-Wheelis commented 6 months ago

Starting to add the survey responses to the public dashboard - want to allow for any survey configuration, including multiple conditional surveys, see discussion in the issue

JGreenlee commented 5 months ago

You may be interested in refactoring this to utilize the survey_prompted_for_trip function I wrote on e-mission-common.

https://github.com/JGreenlee/e-mission-common/blob/b870cf2e1f974d0d9500e9d806b782a5f45ac0eb/src/survey/conditional_surveys.py#L55

I have not yet experimented with adding e-mission-common to one of our Python projects, but it should work.

Abby-Wheelis commented 5 months ago

I have not yet experimented with adding e-mission-common to one of our Python projects, but it should work.

I have done some experimenting today, but been largely unsuccessful, mostly because I can't figure out how to import the repo as a dependency:

I found this: discussion which seemed to lead to the idea that I could add the dependency to the .yml file using pip such as:

   - pip:
     - git+https://github.com/pythonforfacebook/facebook-sdk.git

I tried adding adding the dependency following this format to the .yml file that gets called from the Dockerfile, but I have yet to find a url that does not fail - sometimes the connection times out, I've had errors about GitHub denying the connection, but haven't been able to get it to work yet.

JGreenlee commented 5 months ago

I tried as well and am realizing I need to make some changes for e-mission-common to be recognized as a valid package. I've never created a Python package before, so I didn't know.

I will reply back when I think it's ready

JGreenlee commented 5 months ago

Done https://github.com/JGreenlee/e-mission-common/releases/tag/0.3.2

I got it working in e-mission-server, so I would expect it to work here too.

Abby-Wheelis commented 5 months ago

I was able to install the package, but I'm struggling to import it in a way that I can access the functions I need. @JGreenlee did you push any of the work you did with it on the server anywhere that I could look at?

I've tried from emcommon.survey import conditional_surveys but get ModuleNotFoundError: No module named 'logger'

If you don't already have that work somewhere, don't worry about it! I have time this week so I can try and debug further.

JGreenlee commented 5 months ago

did you push any of the work you did with it on the server anywhere that I could look at?

I thought it was working, but maybe not everything is working. It seems like the problem is with internal imports (between files within emcommon). It would seem that the proper form for those internal imports would be import emcommon.logger as Logger, instead of just import logger as Logger. But doing that makes the JS unable to compile.

I will have to look at it later. The next thing I can think to try is changing the folder structure of e-mission-common from a "flat layout" Python package to a "src layout" package

JGreenlee commented 5 months ago

@Abby-Wheelis Try version 0.4.0 https://github.com/JGreenlee/e-mission-common/releases/tag/0.4.0

Abby-Wheelis commented 5 months ago

I was just able to use the emcommon function to check what survey was prompted for a trip! I'll integrate this fully and remove my code so we can have a single implementation going forward!

Abby-Wheelis commented 4 months ago

Having merged in the changes from #123, the baseline charts - sensed modes, etc - are working, and in stacked bar chart form

The aggregation step is confusing to some degree (I don't think we'll need it here) so I removed it for now, the charts with no data seem to be working OK for now, but I can't get any of the charts with data to show up, they are failing silently

Abby-Wheelis commented 4 months ago

After a fair bit of messing around, got a chart to generate! Still erroring out down the line but here's an example of what a simple question looks like:

image

Abby-Wheelis commented 4 months ago

The error is happening because when I groupby one of the questions, there are two columns by the same name - I think this question appears in both the "return" and "roaming" surveys ... not sure how to handle this or why I wasn't seeing it before

shankari commented 4 months ago

@Abby-Wheelis I think you will also need some more pre-processing even after the ble_sensed_summary is present. You will need to create a single confirmedMode field for each trip by finding the BLE summary mode with the longest distance, similar to the primary mode (aka something like)

        expanded_ct["confirmedMode?.baseMode"] = participant_ct_df.ble_sensed_summary.apply(lambda md: max(md["distance"], key=md["distance"].get))

I haven't yet reviewed e-mission-common to see how something like confirmedMode.baseMode is interpreted in python; it may be that we may have to change that text in the config anyway. We may have to discuss that with @JGreenlee on Monday.

shankari commented 4 months ago

The error is happening because when I groupby one of the questions, there are two columns by the same name - I think this question appears in both the "return" and "roaming" surveys ... not sure how to handle this or why I wasn't seeing it before

You were using value_counts before. Try going back to that and see if it doesn't error for now. if you can get labels and values separate, you can create a dataframe like pd.DataFrame({"label": labels, "value": values}) and pass it into the plotting function.

shankari commented 4 months ago

If you have the same question in two surveys, I think that the pythonic way would be to group by two (or three?!) different columns (e.g. [surveyName, questionName]) and then aggregate. you should then get something like

surveyName1, questionName1 option1 value11
surveyName1, questionName1 option2 value12
...

and can then get df.loc[surveyName] to get all the questions or df_loc[survey_name, questionName1] to get all the labels and values for that bar

Abby-Wheelis commented 4 months ago

If you have the same question in two surveys, I think that the pythonic way would be to group by two (or three?!) different columns (e.g. [surveyName, questionName]) and then aggregate

hmmm, I don't think I have a survey name column right now (can add one) but do we want to show the question split by survey? The concrete example of this that I'm seeing right now is the question "you were aware of the range before you made the trip" - I guess the difference does matter, at least somewhat, I was thinking that it wouldn't but it will make the quality text make more sense, at least

I'm ALSO seeing an error now where it's not recognizing "5.0" in the options dictionary, where we do have "5" - need to patch that up as well

As it's nearly 2am here I'm not sure how much longer I can/should work on this tonight but was really hoping to get it into an at least roughly working shape so we'll see

Abby-Wheelis commented 4 months ago

I merged the columns with the same name and am now grouping by both the survey name and the question. The script runs and produces charts, though I'm not convinced that it's all of the charts - some charts are missing data - which is consistent with what i was seeing before - but I want to check the data since I think it might be missing some/this is new data so I'm not sure what to expect

Abby-Wheelis commented 4 months ago

Ok there are absolutely bugs, seeing errors presenting in two main ways:

  1. there clearly IS data, but it isn't plotting

    Screenshot 2024-05-05 at 2 17 03 AM

  2. data plots with the error text overlain

    Screenshot 2024-05-05 at 2 17 12 AM

this one is even, not showing one of the labels

Maybe it has to do with the labels and their translations and the color mapping? (I've been messing with this to try and get something working)

Abby-Wheelis commented 4 months ago

Maybe it has to do with the labels and their translations and the color mapping? (I've been messing with this to try and get something working)

This is exactly the problem, added print(label in colors.keys()) and all the charts that work have True and all the buggy charts have at least one False

Abby-Wheelis commented 4 months ago

Coming back to this tonight, with the goal of at least understanding what needs to be done still, including known bugs and getting the bluetooth mode/consequences of that change:

Abby-Wheelis commented 4 months ago

I implemented a workaround for the color mapping for now, we will want to keep something like this for the "multiple select" responses, but I think some of the errors were from answers in the data coming from old surveys (and no longer reflected in the data) -- now all the charts generate!

This does reveal a great example of why we should push the legend below the chart itself for the surveys (granted, I distinctly remember I was the one to select all of the options at once to create an edge case, but it is something we should be accounting for regardless) image

shankari commented 4 months ago

@Abby-Wheelis I pushed the BLE matching to staging earlier today, which seemed to work fine, so I just pushed it to fermata as well. I should have a snapshot with the BLE data structures ready in a few hours.

Abby-Wheelis commented 4 months ago

I think some of the errors were from answers in the data coming from old surveys (and no longer reflected in the data)

I checked up on this, and I think I was right - the two charts that needed "filler" colors for labels missing from the map were from answers that are no longer in the surveys as of the config's recent version. Things like i_don_t_know becoming unsure are responsible for the gaps, not some lag in the code! I also have fairly few responses for a lot of the questions - looking into this next, is it from survey churn or the code?

Abby-Wheelis commented 4 months ago

I also have fairly few responses for a lot of the questions - looking into this next, is it from survey churn or the code?

The charts are fine, we just have few responses - there are, in this dataset right now:

I am seeing the correct number of survey responses for each, now feeling confident that the charts are working

Abby-Wheelis commented 4 months ago

I have loaded the new data, and can access the ble_sensed_summary in the confirmed trips - there are 6 trips with a mode sensed as CAR and 330 sensed as UNKNOWN

When I go to use the function from emcommon to determine what survey should have been assigned to a given survey, I get a syntax error when trying to perform the scoped eval: Error evaluating showsIf for survey {'surveyName': 'DfcGasTrip', 'not-filled-in-label': {'en': 'Gas Car Survey'}, 'showsIf': "confirmedMode?.baseMode == 'CAR'"}: invalid syntax (<string>, line 1)

I'm guessing it has to do with the confirmedMode?.baseMode -- maybe I can escape the question mark? After a bit of back and forth I ended up just implementing a ShowsIfPy, which check the confirmedMode_baseMode, this is working now

A lot of the trips have an unknown or even missing baseMode so we only have:

I did notice the number of TripConfirmSurvey responses increased (by 2) - maybe this is from someone not having a recent version of the app? And there were no additional responses for any of the other surveys -- I'm a little concerned that there are only 2 new responses (and presumably to an old survey) between April 24th and May 6th, but maybe people aren't responding? I think I filled out surveys after this time but maybe not

Charts seem to be working well, I tried them on the frontend dashboard but the boxes are still square (from pie charts) will need to fix that

Quality text works - the 0 trips in the denominator are from none of the trips having a detected ECAR image

shankari commented 4 months ago

I have loaded the new data, and can access the ble_sensed_summary in the confirmed trips - there are 6 trips with a mode sensed as CAR and 330 sensed as UNKNOWN

I also noticed this after the pipeline finished, and was suspicious of the results. I also checked and they are all from the same user. But I don't think that should hold up this PR. Let's just remove the quality text for this alpha release of the public dashboard, get it out and then investigate the ble_sensed_summary values at our leisure. In the PR to add the summary, @JGreenlee said that he did see an E_CAR when he ran the changes against a snapshot running on his laptop. So we just need to figure out what is going on by comparing his run against the new snapshot.

Regardless, I don't think that should hold this up. This is working assuming that the data is correct. So we should merge it so that the fermata team can see what it will look like when they have real data. We should then investigate the data and get it to work.

JGreenlee commented 4 months ago

Yes I was getting E_CAR in the summaries of several trips for a user I tested the pipeline on. It wasn't me, but they had a lot of trips. Presumably it was one of the GSA testers' opcodes

shankari commented 4 months ago

I do think that there is something wrong with the data. I checked the UUIDs for bluetooth_ble entries and there are 11 entries. But we only have matched entries from one user. 

>>> len(edb.get_timeseries_db().find({"metadata.key": "background/bluetooth_ble"}).distinct("user_id"))
11

>>> for u in edb.get_timeseries_db().find({"metadata.key": "background/bluetooth_ble"}).distinct("user_id"):
...     print("Number of BLE entries %s" % edb.get_timeseries_db().count_documents({"metadata.key": "background/bluetooth_ble", "user_id": u}))
...     print("Number of location entries %s" % edb.get_timeseries_db().count_documents({"metadata.key": "background/filtered_location", "user_id": u}))
...     print("Number of trips %s" % edb.get_analysis_timeseries_db().count_documents({"metadata.key": "analysis/confirmed_trip", "user_id": u}))
...
Number of BLE entries 806
Number of location entries 2030
Number of trips 0
Number of BLE entries 23
Number of location entries 391
Number of trips 0
Number of BLE entries 13673
Number of location entries 33994
Number of trips 0
Number of BLE entries 33131
Number of location entries 5270
Number of trips 0
Number of BLE entries 84359
Number of location entries 33678
Number of trips 29
Number of BLE entries 2462
Number of location entries 45
Number of trips 0
Number of BLE entries 2050
Number of location entries 1330
Number of trips 0
Number of BLE entries 1579
Number of location entries 3113
Number of trips 0
Number of BLE entries 17
Number of location entries 338
Number of trips 0
Number of BLE entries 16732
Number of location entries 18930
Number of trips 0
Number of BLE entries 2190
Number of location entries 3765
Number of trips 0

There's definitely something wrong. We need to make sure that the pipeline runs correctly when running locally and then see what we can find from the production logs.

Abby-Wheelis commented 4 months ago

Bumped legend below iff the strings are long, in my opinion it's better to have things like (Yes/No) and (1/2/3/4/5) to the right and only push larger labels below, but open to changing this - may see what it looks like formatted into the dashboard and if the inconsistency is too harsh

image image

In the AM: fix formatting on the frontend, clean up code, and then I should be able to test!

Abby-Wheelis commented 4 months ago

Some of the charts are turning out a little squished - the only change I've made is to increase their bounding boxes to 10x4, which seems to be what the other stacked bar charts get, but I'll investigate that further

Screenshot 2024-05-07 at 8 10 40 AM
Abby-Wheelis commented 4 months ago

Starting to test ... most of the baseline charts are failing to generate ... going back to the notebooks to try and reproduce ...

Also failing in the notebooks ... in generic_metrics the only charts that generated were number of trips by day and number of trips by weekday - the rest either have an error displayed or a debug df table, but there is data, shouldn't the charts be showing up with just sensed?

Actually, to answer my own question, maybe not - the sensed mode is different (bluetooth), so we may need a workaround to introduce it ... but we do have the data (see below) - we'll want to update this to be the matched mode, but maybe that can be a next step

Screenshot 2024-05-07 at 11 26 59 AM

So maybe it's the "labels"? We have user input but not traditional labels ...

expanded_ct.groupby("Mode_confirm").agg({distance_col: 'count'} throws aKeyError, which is why we show the debug dataframe ... I bet the single bar could be plotted. Yep, commented out the line to plot the labeled data and the sensed data plots just fine.

So, there is an issue with all the charts that expect labeled data not being able to plot the sensed data because of an error with the labeled data. We have a couple choices here:

If I can do it quickly, I think just debugging is the best plan, given that the bluetooth sensed mode is also buggy (why we took it out of the quality text for now) - else just cut all broken charts for the moment being. @shankari what do you think?

shankari commented 4 months ago

@Abby-Wheelis it might be more meaningful to create a separate notebook for the fleet case. We won't have mode_confirm, and also the sensed mode for the BLE is stored separate (ble_sensed_mode instead of sensed_mode). So both the bars in the standard notebooks would be different. We can just change the index.html to show different charts if it in "fleet mode" We should revisit this once we start incorporating bluetooth into regular OpenPATH, but that should be good enough for an alpha version.

To plot the `ble_sensed_mode,

can we just replace the primary_mode with the primary sensed mode?

Yes, as in, you would need to compute the primary_ble_sensed_mode like we compute the primary mode now, but then you can just use it in the same way.

We may want to also group the bars differently for the fleet case - since we won't have labeled versus confirmed, we may want to do trip count versus distance. So basically only one figure but with two bars.

Abby-Wheelis commented 4 months ago

it might be more meaningful to create a separate notebook for the fleet case. We won't have mode_confirm, and also the sensed mode for the BLE is stored separate (ble_sensed_mode instead of sensed_mode). So both the bars in the standard notebooks would be different. We can just change the index.html to show different charts if it in "fleet mode"

I think I could handle the dfc-fermata changes this way, but it would still be broken for all other survey cases (likewashingtoncommons). Maybe a metrics notebook for survey cases - which handled both fleet and non-fleet cases (difference would be displaying sensed mode or ble sensed mode) would be best?

We may want to also group the bars differently for the fleet case - since we won't have labeled versus confirmed, we may want to do trip count versus distance. So basically only one figure.

I think this would work, we might face duplicating or losing some of the charts, though:

That's fairly minimal loss and fairly minimal duplication, that feels OK to me, will proceed with this

shankari commented 4 months ago

loose trips under 80%

We can retain trips under 80% and land trips as two more figures with count and distance. I am not sure they are as meaningful for the fleet case, but might be for the other survey cases (like washington commons)

Abby-Wheelis commented 4 months ago

Three charts for the sensed modes: count/distance, under 80%, and land modes: image

image

image

We may want to refine the text on these axis/titles in particular - going to pipe through to the frontend first

Abby-Wheelis commented 4 months ago

Testing done with dfc-fermata (May 6th) dataset - can run everything from the command line with no breaking!

When dashboard loads: Screenshot 2024-05-07 at 4 11 47 PM

Non-survey metrics: Screenshot 2024-05-07 at 4 12 23 PM

More survey examples: Screenshot 2024-05-07 at 4 13 04 PM

More info examples: Screenshot 2024-05-07 at 4 29 54 PM

Surveys for May are showing the debug df since there are no responses

Testing with washingtoncommons as well, getting 1 weird behavior: image The last segment is failing to plot for some reason ....

Abby-Wheelis commented 4 months ago

Ah, it was the Other color not present in the map for survey responses - patch incoming

Abby-Wheelis commented 4 months ago

fixed! better

shankari commented 4 months ago

Testing done:

Abby-Wheelis commented 4 months ago

survey responses don't work, but that is because there are only TripConfirm entries in the dataset ?!

What dataset were you using? 336 trips is a lot less than the 990 I have in the May 8th dataset, it it's old enough I would expect only to see TripConfirm survey entries, since I believe that is what we were using for our very preliminary internal testing - then we went to Roam/Return/Gas around the end of March

Abby-Wheelis commented 4 months ago

This broke normal deployments on staging!!!

Screenshot 2024-05-08 at 9 49 21 AM

I'll work on a cleanup for this issue and the points mentioned above