BUG: Dashboard has numpy memory error for large data set

[yim0331]

Describe the bug A clear and concise description of what the bug is. Dashboard data filters do not work with large data set. The dashboard server responds to the 500 error by numpy.core._exceptions._ArrayMemoryError:

To Reproduce Steps to reproduce the behavior: Install tdcosim in python 3 64bit and run the dashboard with 5GB pickle file FYI. the 5GB data file is available D://dashboard/ folder in the testing server machine

Expected behavior A clear and concise description of what you expected to happen. The dashboard data filter is working without error

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. Windows 10]
• PSS/E Version [e.g. 33.3.0] Windows 10, 4Ghz CPU, 32GB memory machine

Additional context Add any other context about the problem here.

[yim0331]

@sibyjackgrove this is the test result

Using Pickle file (default dashboard run):

• File size = 4.8G (Pickle)
• Initiate web server takes 280s
• While loading: memory usage is going up to 20GB (Idle)
• On the peak time (initial rendering of the web page by browser request) => 45GB
• Clicking any filtering option => memory usage is increased 10GB, so a total is 30GB from idle status. The applying time of the option is 6.5s

Using Dask option (using "-d true -o folder_name" options):

• File size = 9.6G (CSV)
• While loading: memory usage is going up to 10G
• On the peak time (initial rendering of the web page by browser request) => 45G
• Clicking any filtering option => memory usage is increased 10GB, so a total is 19GB from idle status. The applying time of the option is 5.39s

[sibyjackgrove]

@yim0331 Thank you for summarizing. I am puzzled by why the peak time in memory usage is so high for Dask.