Fix GUI performance issues with large numbers of variables

[aowen87]

Open sesame.pdb with Try Harder option set to 2. This will load a database with about 3500 mesh objects each with 515 variables defined.The GUI (and pan/zoom ops in the Viewer) can exhibit 1020 second delays between adding plot and having it appear in the plot list, deleting a plot, hiding a plot, etc. The shear number of items is part of the problem. Reducing the number of items can be improved somewhat... don't do automatic expresions (mesh quality, time derivative, vector magnitude). There are prefs. for these. Note: mesh quality exprs only get created for ucd meshes so that pref. has no effect on non-ucd mesh databases don't do operator-created expressions currently know pref. for this so need to add one We could adjust logic in operators to create operator exprs only when mesh or var count is low, say less than 50 don't create subset plot entries for meshes of just one block. Thats pointless and easily corrected. We populate the menu in the viewer too. Do we really need to do that? An informal survey indicated several developers weren't even aware of the option in the viewer's menu suggesting users are even less aware of this feature. I think it would be easiest/best to remove it entirely. Short of that, maybe it is never created until some user actually does the GUI actions to suggest its needed by right-clicking in the viewer, going to Plots and then trying to add a plot. The above changes actually make things almost useable for seseame.pdb (an extreme example of a database with a large number of objects) Next, we need to add smarts to not constantly create and destroy variable menus as frequently as we currently do. We enter VariableMenuPopulator a lot. It has some smarts to return early but I think we need to improve that. I think we should return early when the database and timestep are same since last visit. But, I don't think we do that. Furthermore, some of the logic there to determine early return is expensive (operator== on expression list iterates over all expressions). The order in which we consider each expression's members in an individual expression's operator== could have some impact too. For example, we should try to compare those attributes of expressions that are more likely to be different first. The operator[] on ExpressionList is a linear search. That can be fixed easily too. I've currently coded a simple hashing mechanism there and improved performance quite a bit. We're also taking too much memory. The GUI process easily reaches 1 Gigabyte on sesame.pdb. Hacks for several of the obove changes demonstrate we can get that back down to something reasonable, maybe 250 Mb. However, I think the right long term answer is to switch to an on-demand form of menu creation where we keep a sort of dom object internally regarding what the full menu structure should look like but only realize the parts of that structure with the Qt widgets that are needed at any moment. I rated impact as high but expected use as rare because I think it is rare we deal with databases with large numbers of objects.

-----------------------REDMINE MIGRATION----------------------- This ticket was migrated from Redmine. As such, not all information was able to be captured in the transition. Below is a complete record of the original redmine ticket.

Ticket number: 2824 Status: Resolved Project: VisIt Tracker: Feature Priority: Normal Subject: Fix GUI performance issues with large numbers of variables Assigned to: Mark Miller Category: - Target version: 2.12.3 Author: Mark Miller Start: 05/21/2017 Due date: % Done: 70% Estimated time: 40.00 hours Created: 05/21/2017 08:42 pm Updated: 06/08/2017 08:59 pm Likelihood: Severity: Found in version: 2.12.1 Impact: 5 - Very High Expected Use: 2 - Rare OS: All Support Group: Any Description: Open sesame.pdb with Try Harder option set to 2. This will load a database with about 3500 mesh objects each with 515 variables defined.The GUI (and pan/zoom ops in the Viewer) can exhibit 1020 second delays between adding plot and having it appear in the plot list, deleting a plot, hiding a plot, etc. The shear number of items is part of the problem. Reducing the number of items can be improved somewhat... don't do automatic expresions (mesh quality, time derivative, vector magnitude). There are prefs. for these. Note: mesh quality exprs only get created for ucd meshes so that pref. has no effect on non-ucd mesh databases don't do operator-created expressions currently know pref. for this so need to add one We could adjust logic in operators to create operator exprs only when mesh or var count is low, say less than 50 don't create subset plot entries for meshes of just one block. Thats pointless and easily corrected. We populate the menu in the viewer too. Do we really need to do that? An informal survey indicated several developers weren't even aware of the option in the viewer's menu suggesting users are even less aware of this feature. I think it would be easiest/best to remove it entirely. Short of that, maybe it is never created until some user actually does the GUI actions to suggest its needed by right-clicking in the viewer, going to Plots and then trying to add a plot. The above changes actually make things almost useable for seseame.pdb (an extreme example of a database with a large number of objects) Next, we need to add smarts to not constantly create and destroy variable menus as frequently as we currently do. We enter VariableMenuPopulator a lot. It has some smarts to return early but I think we need to improve that. I think we should return early when the database and timestep are same since last visit. But, I don't think we do that. Furthermore, some of the logic there to determine early return is expensive (operator== on expression list iterates over all expressions). The order in which we consider each expression's members in an individual expression's operator== could have some impact too. For example, we should try to compare those attributes of expressions that are more likely to be different first. The operator[] on ExpressionList is a linear search. That can be fixed easily too. I've currently coded a simple hashing mechanism there and improved performance quite a bit. We're also taking too much memory. The GUI process easily reaches 1 Gigabyte on sesame.pdb. Hacks for several of the obove changes demonstrate we can get that back down to something reasonable, maybe 250 Mb. However, I think the right long term answer is to switch to an on-demand form of menu creation where we keep a sort of dom object internally regarding what the full menu structure should look like but only realize the parts of that structure with the Qt widgets that are needed at any moment. I rated impact as high but expected use as rare because I think it is rare we deal with databases with large numbers of objects.

Comments:

[markcmiller86]

@cyrush this is a key issue related to expression list + GUI performance

[cyrush]

my issue so it's outlined here:

i hit an issue where the gui would refresh the add plots menu N times in a row, where N is the number of expressions I defined via the cli. In my case N = 20, leading to a several second delay before I could add a plot. It was exacerbated by X11 forwarding, but I think its happing all the time.