dres-dev / DRES

Distributed Retrieval Evaluation Server
MIT License
14 stars 3 forks source link

More finegrained viewer update events #342

Open lucaro opened 2 years ago

lucaro commented 2 years ago

The current event structure is to coarse and triggers unnecessary updates. One causes the target view to be reset whenever new scores from a task that requires judging comes in. There should probably be dedicated events for score updates, submission updates and task updates, rather than having this all in one.

lucaro commented 2 years ago

After closer inspection, it turns out that the task update message is already rather specific for events that relate to a change in submissions or scores. The viewer should therefore already be able to properly distinguish what to update, since changes in tasks have a different event. The issue with the reset in the display of the target in case of score changes is therefore a viewer-side one.

sauterl commented 1 year ago

@ppanopticon what is the status on this? I guess the main objective was to not have updates (visually) for instance, when a new task is active in the viewer and judges are still working on their queue within the same evaluation.

ppanopticon commented 1 year ago

I'm not quite sure if this is still an issue after the latest refactoring and what exactly the design goal of this issue is.

lucaro commented 1 year ago

The goal is to avoid superfluous updates. Specifically, the behavior that the playback of the query target of a task is affected by incoming judgments of another task. Task update messages can be ignored if the viewer is currently showing a different task, etc.

ppanopticon commented 1 year ago

This is not something we can handle in the UI without completely rewriting the way these WebSocket messages are being processed in the RunViewer, which I do not have time to do.

So either we leave it as it is or we handle this in the backend, e.g., by not sending a message in updateSubmission() if the ID of the the submission's task does not match the current task.

Generally, it seems questionable to me, if we even need an update in such cases at all.

ppanopticon commented 1 year ago

Based on our discussion of July 3rd, we leave this as it is for now. For future reference, here is the state of the discussion:

Issue

Solution

Conceptually, one should extend the WebSocket messages considerably to support the different constellations. Most importantly, it should have a notion of:

In the viewer, this message payload must be available at all levels of processing. Currently, a message simply triggers the reload of the run state and the message itself is discarded.