Img data json django32

[will-moore]

This is identical to #536, except this is based off v5.22.1 and so is compatible with Django 3.2.

Performance testing

NB: this PR includes TEMP A/B testing

• A) Images with EVEN numbered IDs: - we check the size of the image and only load rendering engine for resolution levels if it's BIG
• B) images with ODD numbered IDs - we ALWAYS load the rendering engine to check re.requiresPixelsPyramid(), if so, we also load resolution levels.

I would expect A (EVEN numbers) to be faster for small images BUT it seems that B is faster?!?! Possibly because the size threshold is using config service to get the BIG image limit.

Memo file requirement

I had hoped that by checking image size, small images (EVEN IDs) would NOT need to read memo file in order to load imgData (required for iviewer right panel).

BUT, seems that the imgData won't load for e.g. idr0090 images where memo file is missing. e.g. https://idr-testing.openmicroscopy.org/webclient/imgData/12542998/ (504 timeout).

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#468. See the console output for more details. Possible conflicts:

• PR #531 knabar 'Removal of python-future compatibility code '
  • omeroweb/webgateway/marshal.py
  • omeroweb/webgateway/util.py
• PR #534 will-moore 'get strings from omero.cmd.ERR and omero.CmdError correctly'
  • omeroweb/webgateway/marshal.py
• PR #536 will-moore 'imgData JSON with no stateful services'
  • omeroweb/webgateway/marshal.py
  • omeroweb/webgateway/util.py

--conflicts

[will-moore]

Testing with this PR deployed on idr-testing definitely felt faster to load right panel of iviewer. Closing now.

[will-moore]

Redeploying the A/B/C testing on idr-testing for more perf testing...

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo /opt/omero/web/venv3/bin/pip uninstall -y omero-web && sudo /opt/omero/web/venv3/bin/pip install git+https://github.com/ome/omero-web.git@refs/pull/537/head && sudo service omero-web restart"; done

[will-moore]

Summary of performance testing...

In the chart below, each green bar is the http response time. Blue (where shown - 1st 2 sections only) is the time to load rendering engine as reported in the JSON response with the ABC testing enabled for this PR.

Testing in 4 sessions below:

• 1st session: idr0010 with this PR - A/B/C testing in a bit of random order
• 2nd session: idr0090 with this PR - A/B/C testing in order - e.g. every 3rd image uses rendering engine (blue bars)
• 3rd session: idr0090 without this PR - no A/B/C testing - all bars are the same
• 4th session: idr0010 without this PR - no A/B/C testing - all bars are the same

Conclusions:

• For idr0010, this PR makes very little (no discernible) difference. Times to load rendering engine to get pyramid levels are about 0.04 secs. Probably about the same for other stateful services: insignificant compared to overall http response times of 0.5 secs
• For idr0090, loading rendering engine to get pyramid levels takes about 0.5 seconds.
• Even when we do load RE (for pyramid levels) in this PR, http response is about 1.4 secs, which is faster than without this PR (about 1.9 secs) suggesting that for idr0090, rawPixelsStore for pixels min/max and renderingEngine loading of rendering settings together add an extra 0.5 secs in total (for idr0090).

To summarise:

• Don't expect this PR to improve performance for most data in OMERO
• Even if we always use RenderingEngine to test/load the pyramid levels, this PR still has a performance improvement for large (slow) data like idr0090