Azure HDInsight Spark lab Jupyter Notebook error

[tomeeraerts]

When visualizing data after a sql command, you get an error

%%sql
show tables
Error is:
AttributeErrorTraceback (most recent call last)
/usr/bin/anaconda/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
    902                 pass
    903             else:
--> 904                 printer(obj)
    905                 return True
    906             # Finally look for special method names

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in display_dataframe(df)
    114 
    115 def display_dataframe(df):
--> 116     selected_x = select_x(df)
    117     selected_y = select_y(df, selected_x)
    118     encoding = Encoding(chart_type=Encoding.chart_type_table, x=selected_x, y=selected_y,

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in select_x(data, order)
     70         _validate_custom_order(order)
     71 
---> 72     d = _classify_data_by_type(data, order)
     73 
     74     chosen_x = None

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in _classify_data_by_type(data, order, skip)
     48     for column_name in data:
     49         if column_name not in skip:
---> 50             typ = infer_vegalite_type(data[column_name])
     51             d[typ].append(column_name)
     52 

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in infer_vegalite_type(data)
     14     """
     15 
---> 16     typ = pd.api.types.infer_dtype(data)
     17 
     18     if typ in ['floating', 'mixed-integer-float', 'integer',

AttributeError: 'module' object has no attribute 'api'

This is because an outdated library is used by the notebook. It can be fixed by running

sudo -HE /usr/bin/anaconda/bin/conda install pandas

on the cluster. After that the different options for visualizing data are available again.

This should be added to lab to make the experience as smooth as possible.