Python Performance concern - Multiple methods reading same Pandas Data Frame from a particular Pickle file

[RohitDhankar]

For creating summary statistics charts / plots from Bokeh and Holoviews , following steps are followed -

1/ Data set , which is chosen by the Data Analyst (end user) is culled out from the PostgreSQL DB . Lets presume we limit this data sets size to 500 rows of data .

2/ This smaller set of 500 rows - lets call it df_eda [Data frame for Exploratory data analysis ] , is then 'pickled' and stored / persisted - onto the server local drive. This pickled data set is named - 'df_holoviewPlots.pkl' . Its referred here in our code - (https://github.com/RohitDhankar/DigitalCognition/blob/17cb9e2781253f3635b46b9cc038baa382874443/dc_dash/dc_holoviews.py#L48)

3/ Reason's we pickle and persist 'df_eda' ? Whenever plots are rendered - they will need 'df_eda' as input parameter. We have no control over how many times , user wants to see Charts/ Plots .

Thus over the top - these options available , as on 11th June 19 , they all need to be performance tested -

• read 'df_eda' from the MYSQL DB / PostgreSQL directly [LEAST PREFERRED ], or
• read from a Pickle / cPickle - file on Local DIR , or
• hold 'df_eda' in server/ system RAM , or
• store 'df_eda' in a , in-memory cache DB like - 'REDIS'
• store 'df_eda' as JSON or XML - file on Local DIR , in place of Pickle / cPickle etc.
• store 'df_eda' as JSON within browser on clientside - in place on Local DIR serverside ,

Have raised this Python Performance concern here on SO - Multiple methods reading same Pandas Data Frame from a particular Pickle file

[RohitDhankar]

ClientSide - Session Storage of 'df_eda' in JSON format looks promising - need to performance test options.

[RohitDhankar]

As of now - using the localStorage.setItem('myObj_bokeh'... , will maybe do perf. tests for other scenarios later...

https://github.com/RohitDhankar/DigitalCognition/blob/307c3262def2926cee6a17c332a2f604a77b0b86/dc_dash/templates/dc_dash/eda_sidebar.html#L1320