Performance of loading long batch files in SANS GUI

[rbauststfc]

If you load a long batch file (i.e. approximately 50 lines or longer) into the SANS GUI it is very slow and the GUI usually becomes temporarily unresponsive. Having loaded the file, if you try to change any of the contents of the table via the interface this is also very slow (similar behaviour as when loading). This is the case even when used on IDAaaS.

SANS scientists have confirmed that they experience this but have just gotten used to it as part of their workflow. However, it is obviously not ideal. The GUI gives no indication that it is still working on the load request, so the long pause where nothing is happening can be confusing. Scientists generally don't experience the slow speed when a value is changed directly in the GUI table as they tend to change values by editing the batch file and re-uploading. We have, however, had a report of a user attempting to do this and Mantid froze and ultimately crashed. We cannot seem to replicate the crash that the user saw, despite testing with the relevant batch file and user file, although the loading is very slow and this makes it more likely to freeze as a result.

We could do with investigating the underlying cause of the slow performance when loading batch files and editing the GUI runs table, and identify if there is anything we can do to improve performance and/or usability here. After a very quick initial look at the batch loading code, we seem to look up each run (I think maybe just the sample scatter run, but would need to confirm) to get metadata from the data file (i.e. things like geometry information). This doesn't seem to be done for validation purposes - nothing happens if the file isn't found - it just seems to be to look up metadata. This would definitely explain why the load is so slow as file finding is known to be very slow. We've also had reports following SANS Group performance testing that if you "optimise" the data search paths (i.e. make sure the one you're using for the reduction is the first that will be searched) then it considerably improves performance, which further supports that the file finding is the underlying cause.

Time estimate on this issue is for investigation of the underlying cause and assessment of what should be done. It's not yet possible to estimate the time required to implement any required solution.

[smk78]

Of possible relation here is this check box:

What exactly this does (or not) would be worth investigating as part of this issue.

[rbauststfc]

On the use optimizations checkbox, it would be interesting to double check it's not used elsewhere, but I found during a recent investigation that this setting is used in the main reduction workflow to determine whether or not to attempt to re-use an already reduced can from the workspace list rather than unnecessarily reducing it again (see the do_reduction method of SANSingleReduction.py). In the tooltip, the reference to the long batch file is because when using this setting we keep some workspaces in the list that would otherwise be removed (to allow them to be re-used) and so if the batch file is very long it could start to fill up the memory. This use of the setting only has an impact when the reduction is being run, so shouldn't impact on simply loading the batch file into the GUI. We can double check as part of this issue that this setting is not being used anywhere else in the codebase.