`Missing_dict` doesn't appear to be working properly

[shankari]

Got the following error while trying to generate entries for February. Clearly we don't have a mapping for gasolinera, but I thought that should be handled by the missing_dict code, at least for now.

$ sudo run_from_host/generate_plots.sh generic_metrics_ebike_project.ipynb prepilot -d 2021 02
Traceback (most recent call last):
  File "bin/generate_plots.py", line 35, in <module>
    nbclient.execute(new_nb)
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/client.py", line 1087, in execute
    return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/client.py", line 541, in async_execute
    cell, index, execution_count=self.code_cells_executed + 1
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/client.py", line 832, in async_execute_cell
    self._check_raise_for_error(cell, exec_reply)
  File "/root/miniconda-4.8.3/envs/emission/lib/python3.7/site-packages/nbclient/client.py", line 740, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
labels_tp = expanded_ct['Trip_purpose'].value_counts(dropna=True).keys().tolist()
values_tp = expanded_ct['Trip_purpose'].value_counts(dropna=True).tolist()
plot_title="Number of trips for each purposes (selected by users)\n%s" % quality_text
file_name= 'ntrips_purpose%s.png' % file_suffix
pie_chart_purpose(plot_title,labels_tp,values_tp,file_name)
print(expanded_ct['Trip_purpose'].value_counts(dropna=True))
------------------

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-18-44b09bc13a9a> in <module>
      3 plot_title="Number of trips for each purposes (selected by users)\n%s" % quality_text
      4 file_name= 'ntrips_purpose%s.png' % file_suffix
----> 5 pie_chart_purpose(plot_title,labels_tp,values_tp,file_name)
      6 print(expanded_ct['Trip_purpose'].value_counts(dropna=True))

/usr/src/app/saved-notebooks/plots.py in pie_chart_purpose(plot_title, labels, values, file_name)
    120     wedges, texts, autotexts = ax.pie(m_values,
    121                                       labels = m_labels,
--> 122                                       colors=[colours[key] for key in labels],
    123                                       pctdistance=0.85,
    124                                       autopct=lambda pct: func(pct, values),

/usr/src/app/saved-notebooks/plots.py in <listcomp>(.0)
    120     wedges, texts, autotexts = ax.pie(m_values,
    121                                       labels = m_labels,
--> 122                                       colors=[colours[key] for key in labels],
    123                                       pctdistance=0.85,
    124                                       autopct=lambda pct: func(pct, values),

KeyError: 'gasolinera'
KeyError: 'gasolinera'

[shankari]

@start0318 can you test with an actual missing value to ensure that it works? You can do this by removing some of the values from the existing purpose dict.

[shankari]

This also happens with the current generic_metrics_ebike_project notebook. I suspect we weren't running into this earlier because we were only plotting the top 10 purposes. Now that we dynamically merge the small entries, we are running into this.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-18-44b09bc13a9a> in <module>
      3 plot_title="Number of trips for each purposes (selected by users)\n%s" % quality_text
      4 file_name= 'ntrips_purpose%s.png' % file_suffix
----> 5 pie_chart_purpose(plot_title,labels_tp,values_tp,file_name)
      6 print(expanded_ct['Trip_purpose'].value_counts(dropna=True))

/usr/src/app/saved-notebooks/plots.py in pie_chart_purpose(plot_title, labels, values, file_name)
    120     wedges, texts, autotexts = ax.pie(m_values,
    121                                       labels = m_labels,
--> 122                                       colors=[colours[key] for key in labels],
    123                                       pctdistance=0.85,
    124                                       autopct=lambda pct: func(pct, values),

/usr/src/app/saved-notebooks/plots.py in <listcomp>(.0)
    120     wedges, texts, autotexts = ax.pie(m_values,
    121                                       labels = m_labels,
--> 122                                       colors=[colours[key] for key in labels],
    123                                       pctdistance=0.85,
    124                                       autopct=lambda pct: func(pct, values),

KeyError: 'gas'

[start0318]

Fixed this with these two lines in notebooks

convert a dictionary to a defaultdict

dic_pur = defaultdict(lambda: 'Other',dic_pur) dic_re = defaultdict(lambda: 'Other',dic_re)

[shankari]

closed by #22