app crashes when selected catalog is too big

[prjemian]

Here is an exception trace when it crashed for me:

Traceback (most recent call last):
  File "/home/beams1/JEMIAN/Documents/projects/BCDA-APS/gemviz23/gemviz23/demo/resultwindow.py", line 170, in displayTable
    data_model = TableModel(self.cat)
  File "/home/beams1/JEMIAN/Documents/projects/BCDA-APS/gemviz23/gemviz23/demo/resultwindow.py", line 78, in __init__
    self._uidList = list(reversed(list(self._data)))
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/tiled/client/node.py", line 232, in __iter__
    content = self.context.get_json(
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/tiled/client/context.py", line 595, in get_json
    self.get_content(
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/tiled/client/context.py", line 557, in get_content
    handle_error(response)
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/tiled/client/utils.py", line 29, in handle_error
    response.raise_for_status()
  File "/home/beams/JEMIAN/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://otz.xray.aps.anl.gov:8000/api/v1/node/search/4id_polar?page%5Boffset%5D=5200&page%5Blimit%5D=100&fields=&sort=time'
For more information check: https://httpstatuses.com/500
Aborted (core dumped)
(base) jemian@otz .../BCDA-APS/gemviz23 $

[prjemian]

Aha! It crashed while we are building the list of uids.

  File "/home/beams1/JEMIAN/Documents/projects/BCDA-APS/gemviz23/gemviz23/demo/resultwindow.py", line 78, in __init__
    self._uidList = list(reversed(list(self._data)))

We can avoid this crash by using a pager, to view only n runs at a time. At most 1,000 at a time, reasonable choices might be: 10, 25, 50, 100, 500, 1000. The above exception shows the crash occurred at offset 5200 as it was trying to get 100 more runs from the server. (page[offset]=5200&page[limit]=100)

[prjemian]

That will help make the load times faster, by limiting how many runs are requested at a time. Right now, it takes about 10 seconds (or so, estimate only) before the program crashes.

[prjemian]

We'll need a row of push buttons in the results window, below the QTableView`, to adjust the page (the specific runs) in view. The row should also have a QComboBox to select the number of runs to be paged.

| Scan ID | Plan Name | Detectors | Positioners |
|---------|-----------|-----------|-------------|
...
|---------|-----------|-----------|-------------|

  |<     <     [combo]^    >    >|

We can modify the row numbers shown in the headerData() method to include the "page" offset, as it is changed when the user presses the push buttons.

button	name	meaning
\|<	first	start table with oldest run
<	back	move one page backward (toward oldest run)
[combo]^	pageSize	QComboBox to pick the page "size" (number of runs to show in the table)
>	next	move one page forward (toward oldest run)
>\|	last	end table with last row

[prjemian]

While this line is not a pager, it limits the number of runs to at most 500, which could be useful during development until the pager is working:

self._uidList = list(reversed(list(self._data.keys().tail(500))))

[prjemian]

With the pager terms of offset & pageSize and after inspection of the tiled docs & source code, the _key_slice() method is a much more efficient way to grab the list of run UIDs to build the table. Here is an example:

def get_keys(cat, offset=0, size=10, ascending_time=True):
    return cat._keys_slice(offset, offset + size, 1 if ascending_time else -1)

[prjemian]

Replace: https://github.com/BCDA-APS/tiled-viz2023/blob/d87328d8ca2cdb175b91bac7126e6d422a0420c3/gemviz23/demo/resultwindow.py#L78

with:

        self._uidList = self._data._keys_slice(offset, offset + pageSize, 1 if ascending_time else -1)

We'll need other modifications to handle changes to the new parameters: offset, pageSize, and ascending_time. Also to enable computation of offset and pageSize from the view's new QPushButtons and QComboBox.

If we want to change sorting or query parameters, then we need to use self._data.new_variation(queries=queries, sorting=sorting). Here's an example:

def get_keys(cat, offset=0, size=100, queries=None, sorting=None):
    sorting = sorting or [("time", -1)]  # default is newest to oldest
    rcat = cat.new_variation(queries=queries, sorting=sorting)
    return rcat.keys()[offset:offset + size]

and call it with the custom sorting terms:

        self._uidList = get_keys(self._data, offset=25, size=6, sorting=[("time", -1)])

[prjemian]

BTW, the rowCount() method should return len(self._uidList) (the number of runs to be shown) instead of len(self._data) (the total number of runs in the catalog).

now: https://github.com/BCDA-APS/tiled-viz2023/blob/d87328d8ca2cdb175b91bac7126e6d422a0420c3/gemviz23/demo/resultwindow.py#L83

revised:

        value = len(self._uidList) 

[prjemian]

The simplest explanation for these changes is to push the job of sorting and slicing (use of offset & size) to the tiled server. The local Python code should only manage the selection of offset and size.

[prjemian]

Issue #51 looks like a possible duplicate but identifies that the tiled package has newer releases that might assist. Accept the code changes here as-is and leave #51 for later.

For now, avoid demo with big (more than a few hundred runs) catalogs.