Optimize the prioritytarget view

[rachel3834]

Feedback from Jashandeep:

Digging into the /prioritytargets/ view, it does look like there's some significant processing (time-wise) that takes place after the query: https://github.com/LCOGT/mop/blob/aca606d89a22ec84bb307299e9b01746e3cec719/mop/views.py#L192

Repackage the two lists to extract the parameters to display in the table.

There's probably some optimizations here that could speed things up. For instance doing DB look-up in a for loop is going to be slow: https://github.com/LCOGT/mop/blob/aca606d89a22ec84bb307299e9b01746e3cec719/mop/views.py#L216 target = Target.objects.filter(pk=target_id[0])[0]

[rachel3834]

For the /targets// page, these lines should be moved to the conditional, if possible: https://github.com/LCOGT/mop/blob/aca606d89a22ec84bb307299e9b01746e3cec719/mop/views.py#L37-L38 views.py target = self.get_object() set_target_sky_location(target) https://github.com/[LCOGT/mop](https://github.com/LCOGT/mop)|LCOGT/mopLCOGT/mop | Added by GitHub 9:39 That function does a DB save every call: https://github.com/LCOGT/mop/blob/aca606d89a22ec84bb307299e9b01746e3cec719/mop/toolbox/TAP.py#L108 TAP.py def set_target_sky_location(target): https://github.com/[LCOGT/mop](https://github.com/LCOGT/mop)|LCOGT/mopLCOGT/mop | Added by GitHub 9:40 Which can't be good for performance

[rachel3834]

I've implemented the proposed optimizations as described above so that we are not doing as many DB queries in loops.

[rachel3834]

Timing data suggests this may roughly half the load time for the page.

[rachel3834]

Checkpointing the existing code before further optimizations:

PRIORITYTARGETS started context get 2024-02-22 22:07:33.620458
CHECKPOINT: N DB connections: 1, memory: 248.02MiB
Priority querysets initially find 2 stellar candidate events and 19 BH candidate events
PRIORITYTARGETS took 0:00:00.114046
PRIORITYTARGETS started extract at 2024-02-22 22:07:33.734215
CHECKPOINT: N DB connections: 11, memory: 249.38MiB
PRIORITYTARGETS finished data extract at 2024-02-22 22:07:34.097975
PRIORITY TARGETS extract function took 0:00:00.363760
CHECKPOINT: N DB connections: 105, memory: 250.5MiB
PRIORITYTARGETS started extract at 2024-02-22 22:07:34.098766
CHECKPOINT: N DB connections: 105, memory: 250.5MiB
PRIORITYTARGETS finished data extract at 2024-02-22 22:07:36.690095
PRIORITY TARGETS extract function took 0:00:02.591329
CHECKPOINT: N DB connections: 697, memory: 255.98MiB
After filtering priority targets, returning 2 stellar candidate events and 12 BH candidate events
PRIORITYTARGETS finished context get 2024-02-22 22:07:36.691680 took 0:00:03.071222
CHECKPOINT: N DB connections: 697, memory: 255.98MiB
FINISHED GET_CONTEXT 2024-02-22 22:07:37.105672
CHECKPOINT: N DB connections: 697, memory: 270.97MiB

[rachel3834]

Refactoring using MicrolensingEvents gives results:

PRIORITYTARGETS started context get 2024-02-22 22:58:52.734901
CHECKPOINT: N DB connections: 1, memory: 248.01MiB
/data/software/mop_venv/lib/python3.10/site-packages/django/views/generic/list.py:91: UnorderedObjectListWarning:

Pagination may yield inconsistent results with an unordered object_list: <class 'tom_targets.models.TargetExtra'> QuerySet.

QueryTools: Got 2 targets above priority threshold, 2204 targets outside the HCZ, 4111 targets classified as microlensing, 164 targets not YSOs, 164 targets not QSOs, 164 targets not galaxies
QueryTools: identified 0 targets
QueryTools: Got 19 targets above priority threshold, 2204 targets outside the HCZ, 4111 targets classified as microlensing, 164 targets not YSOs, 164 targets not QSOs, 164 targets not galaxies
QueryTools: identified 1 targets
Priority querysets initially find 0 stellar candidate events and 1 BH candidate events
PRIORITYTARGETS took 0:00:01.877117
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
PRIORITYTARGETS started extract at 2024-02-22 22:58:54.613215
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
queryTools: Retrieved associated data for 0 Targets
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
queryTools: Time taken: 0:00:00.001425
queryTools: collating data on microlensing event set
queryTools: Collated data for 0 targets in 0:00:00.001124
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
PRIORITYTARGETS finished data extract at 2024-02-22 22:58:54.617698
PRIORITY TARGETS extract function took 0:00:00.004483
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
PRIORITYTARGETS started extract at 2024-02-22 22:58:54.619018
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
queryTools: Retrieved associated data for 1 Targets
CHECKPOINT: N DB connections: 79, memory: 269.11MiB
queryTools: Time taken: 0:00:00.001066
queryTools: collating data on microlensing event set
queryTools: collated data for target Gaia23aiy, 0 out of 1
CHECKPOINT: N DB connections: 85, memory: 286.29MiB
queryTools: Collated data for 1 targets in 0:00:00.623020
CHECKPOINT: N DB connections: 85, memory: 286.29MiB
PRIORITYTARGETS finished data extract at 2024-02-22 22:58:55.244476
PRIORITY TARGETS extract function took 0:00:00.625458
CHECKPOINT: N DB connections: 85, memory: 286.29MiB
After filtering priority targets, returning 0 stellar candidate events and 1 BH candidate events
PRIORITYTARGETS finished context get 2024-02-22 22:58:55.244859 took 0:00:02.509958
CHECKPOINT: N DB connections: 85, memory: 286.29MiB
FINISHED GET_CONTEXT 2024-02-22 22:58:55.619288
CHECKPOINT: N DB connections: 85, memory: 281.54MiB

[rachel3834]

Revised code is more efficient on querysets and a bit quicker. Major changes include properly searching for targets that are classified as microlensing, alive, not QSOs etc instead of doing this filtering in a later for loop.
I've added checkpoints so I can gauge how this behaves in operation.

[rachel3834]

A significant issue with this refactor is that all non-Gaia targets have disappeared from the board. This is addressed in Issue #141