Need to display demographic table

[shankari]

We should either combine the demographics with the user table or display it as a separate table in the admin dashboard. With the integrated enketo demographic survey, there is really no other option for deployers to download demographic data.

[shankari]

Starting from durham (this is not true for all_ceo), we store the demographic data as a new manual stream in our timeseries database. Every time the user edits the survey, we create a new entry with a new write timestamp so theoretically we can see how people's demographics change over time (if they are diligent and fill out the survey again).

As you can see from entry.py, this is stored as

            # demographic survey
            "manual/demographic_survey": "onetimesurvey",

So you should be able to query for it it using the abstract timeseries (emission.storage.timeseries.abstract_timeseries). There are examples indb_utils` of querying these.

You likely want to get the aggregate timeseries (esta.TimeSeries.get_aggregate_timeseries()) so you can retrieve demographic data for all users and callfind_entries` on that.

~If you do get a list of users instead and iterate over them , you can probably use get_first_value_for_field with the field of jsonResponse, sorted by metadata.write_ts in DESCENDING order (so you get the most recent entry.~

You should get the jsonResponse from each entry and use something like pd.json_normalize to convert into a dataframe. You can then return the dataframe to a new tab in the "data" page and allow people to download the demographic table as well (like they can download trip table).

The tables automatically handle pagination.

Once you are done with this, please submit a PR so we can make incremental progress. While polishing, we may want to have buttons to allow users to only show (and download) the most recent entry for user.

[shankari]

For the record, we already have a script (that I had linked to earlier) when asking @AlirezaRa94 to create the data tables. bin/debug/export_participants_trips_csv.py in the e-mission-server repo already has code to read the demographic survey and print it out. You just need to change it from using the raw mongodb interface in edb to the timeseries interface as outlined above.

[achasmita]

I have added a new demographics tab in the data page of op-admin-dashboard. I retrieved demographic data by utilizing aggregate timeseries esta.TimeSeries.get_aggregate_timeseries() as you mentioned above.

Now the tab is working with few columns, Also it allows users to download the demographic table as well. I am unsure whether we have to display all the columns listed in data and wondering if we have to format any columns @shankari . As there are many survey columns like 'data.jsonDocResponse.aSfdnWs9LE6q8YEF7u9n85.group_hg4zz25.What_is_your_race_ethnicity',

After displaying right columns I will clean the code and send PR.

[shankari]

As there are many survey columns like 'data.jsonDocResponse.aSfdnWs9LE6q8YEF7u9n85.group_hg4zz25.What_is_your_race_ethnicity',

I would like to see what this looks like in a screenshot, but at a high level: Yes. That represents the question: "What is your race and ethnicity"?

If it is too long to display, we can, as a temporary solution, strip out everything other than the part after the last . before display. So change the column name to display What_is_your_race_ethnicity?

Although I have sent the csv generated by script export_participants_trips_csv to at least two partners and they haven't complained.

The long-term fix might be to actually parse out the XML representing the survey and map the column headers back to the English text. But that is not a priority at this point.

[achasmita]

I have sent you a draft pull request #63

Changes made:

op-admin-dashboard/README.md

Replaced

docker compose -f docker-compose-dash-app.yml build 

docker compose -f docker-compose-dash-app.yml up

with

 docker compose -f docker-compose-dev.yml build

docker compose -f docker-compose-dev.yml up

op-admin-dashboard/docker

Added a shell script (load_mongodump.sh) to load data.

*op-admin-dashboard/.py**

Added a new tab "Demographics" to the data page to display demographic table. Here is the image for demographic table in the data page of op-admin-dashboard.

[shankari]

Fixed in https://github.com/e-mission/op-admin-dashboard/pull/63