#62: Speed up file upload

NexGenAnalytics / WorkVisualizer

High-level, interactive, visual performance analysis for High-Performance Computing (HPC)

https://www.workvisualizer.com/

Other

4 stars 0 forks source link

#62: Speed up file upload #67

Closed cwschilly closed 1 month ago

cwschilly commented 3 months ago

Fixes #62 Fixes #70

Current solution: distribute file upload, wait until everything is uploaded, then distribute processing. Outcome: MD 250 case uploads & processes in 65 seconds

More substantial improvements/changes will need to be made in Phase 2 to scale the app well.

Note: The investigation of streaming vs. writing has been moved to PR #72

cwschilly commented 1 month ago

There's a bug in the slicing analysis pipeline. To recreate, try to show the slice lines on the Events Plot

pierrepebay commented 1 month ago

@cwschilly I was able to replicate the bug with the 120 rank CabanaMD data, not the 4 rank data/cali/md_.... Pushing a fix now, I'll let you check if it works on some larger scale data as well (I don't have the compute available for that right now :)).

pierrepebay commented 1 month ago

@cwschilly I'm also pushing the updated package-lock.json, after running npm audit fix, in regards to #70.