🚑 Displaying the demographic table is generating an error

[shankari]

Report from a partner:

We are facing a small issue with the NREL dashboard. The demographics tab isn't loading (instead it shows the data from the last tab that was selected). Do you know what could be causing this issue?

On checking the logs, I see

ERROR:app_sidebar_collapsible:Exception on /_dash-update-component [POST]
Traceback (most recent call last):
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/dash/dash.py", line 1283, in dispatch
ctx.run(
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/dash/_callback.py", line 450, in add_context
output_value = func(*func_args, **func_kwargs)  # %% callback invoked %%
File "/usr/src/app/pages/data.py", line 79, in render_content
data = store_demographics["data"]
KeyError: 'data'

[shankari]

This is super weird because the demographics is read the same way as the other values

    elif tab == 'tab-demographics-datatable':
        data = store_demographics["data"]
        columns = list(data[0].keys())
        has_perm = perm_utils.has_permission('data_demographics')

[shankari]

confirming that this is not a problem on staging

[shankari]

so the data is filled in here very straightforwardly

@app.callback(
    Output("store-demographics", "data"),
    Input('date-picker', 'start_date'),
    Input('date-picker', 'end_date'),
)
def update_store_demographics(start_date, end_date):
    df = query_demographics()
    records = df.to_dict("records")

    store = {
        "data": records,
        "length": len(records),
    }
    return store

Although it is initialized without the data: field

        dcc.Store(id='store-demographics', data= {}),

My guess is that:

1. either there is an earlier error on load, or
2. the load is taking so long that the app is refreshed before it is complete

If it is (2), the fix is simply to initialize with {"data": pd.DataFrame()} instead

[shankari]

Aha! This is a regression caused by https://github.com/e-mission/op-admin-dashboard/pull/83 That prints out a set of hardcoded entries, including Which_one_below_describe_you_b However, the custom onboarding survey used in this partner's project does not have that column.

So we get

The above exception was the direct cause of the following exception:
--
Traceback (most recent call last):
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/dash/dash.py", line 1283, in dispatch
ctx.run(
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/dash/_callback.py", line 450, in add_context
output_value = func(*func_args, **func_kwargs)  # %% callback invoked %%
File "/usr/src/app/app_sidebar_collapsible.py", line 165, in update_store_demographics
df = query_demographics()
File "/usr/src/app/utils/db_utils.py", line 124, in query_demographics
logging.debug("The data in column is %s" %df["Which_one_below_describe_you_b"])
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/pandas/core/frame.py", line 3807, in __getitem__
indexer = self.columns.get_loc(key)
File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
raise KeyError(key) from err
KeyError: 'Which_one_below_describe_you_b'

Reverting that commit since @achasmita has been able to now reproduce the error in her dev environment