Only predict on all documents when in Active Learning Mode

[jcklie]

Is your feature request related to a problem? Please describe. Right now, recommender predict on all documents while the user only sees one, except when using active learning. These mostly unseen predictions are unnecessary and slow.

Describe the solution you'd like Only run predictions on all documents when in active learning mode.

Additional context This is especially relevant for slow recommenders, e.g. neural external ones.

[reckart]

The core recommender code is not aware of active learning. How would you maintain this decoupling while implementing this feature?

[jcklie]

I would put the information which mode to run maybe in the recommenderState and expose it via the recommender service.

[munterkalmsteiner]

The more documents exist in a project, the larger is the performance hit. See also #1335. I would like to address this, but I'm not sure, how. Wouldn't the PredictionTask need to know which document the user is currently viewing / viewing next? Wouldn't that not also need that the current viewed document is updated once the calculations are complete, to make sense? Currently, as far as I can see, the predictions do not affect the currently viewed document as the task execution is asynchronous and the UI is anyway not updated (I've seen some websocket patch, is that adressing that?). I have the feeling this change has a rat-tail of implications (or better, a stack of pre-requisites to make it work in practice).

[reckart]

Wouldn't the PredictionTask need to know which document the user is currently viewing / viewing next?

Yes. I guess it would.

Wouldn't that not also need that the current viewed document is updated once the calculations are complete, to make sense? Currently, as far as I can see, the predictions do not affect the currently viewed document as the task execution is asynchronous and the UI is anyway not updated (I've seen some websocket patch, is that adressing that?).

We intentionally do not update the UI when predictions are done to avoid the UI "flickering" around. E.g. consider this:

• the user makes an annotation
• the background train/prediciton runs
• the user moves to mouse somewhere to create another annotation
• before the user manages to do that, a websocket update kicks in and changes the layout
• the user clicks somewhere she didn't actually want to click and potentially does something she didn't want to do

For that reason, we include new predictions only in updates to the UI which are triggered by a users's action anyway.

If we implement the present improvement, it could lead to a delay in the prediction display. E.g. if a user switches to another document, the user might see outdated or even no suggestions there because the suggestions have not been updated / produced yet. This is similar to the situation we face when the user is opening a document in a project for the first time - here it may also take a moment before the background predictions have run and can be displayed (on the next user interaction).

[munterkalmsteiner]

If we implement the present improvement, it could lead to a delay in the prediction display. E.g. if a user switches to another document, the user might see outdated or even no suggestions there because the suggestions have not been updated / produced yet. This is similar to the situation we face when the user is opening a document in a project for the first time - here it may also take a moment before the background predictions have run and can be displayed (on the next user interaction).

My observation in the current implementation is exactly that: annotations the user makes on document n are not reflected in the shown suggestions when the user is at document n+1. Currently, also no suggestions are shown when the user opens document 1 (as the prediction may still going on, on documents 2..n, which are irrelevant for the user when looking at document 1).

What I would expect as an annotator is that my annotations are used for training and prediction, once I navigate from the current document away and the new document contains suggestions based on the updated predictions. This would mean that I cannot move to another document until the training and predictions are done. However, if training would only be done if the model has changed, predictions would only be calculated for the relevant document, and the user iterface would signal that the training/prediction cycle is ongoing, the user experience would be better than the current situation as it signals the user what is happening.

[reckart]

This would mean that I cannot move to another document until the training and predictions are done.

A user should be able to access any document also to edit any document, even if the recommenders have not produced suggestions on it yet.

However, if training would only be done if the model has changed, predictions would only be calculated for the relevant document, and the user iterface would signal that the training/prediction cycle is ongoing, the user experience would be better than the current situation as it signals the user what is happening.

Training happens when a) when the user comes to the annotation page and opens some document (not when switching between documents on the page) and b) when the user adds/updates/deleted any annotation.

The work towards incorporating websocket support is done in order to allow us providing real-time information to the user about the status of background jobs, in particular whether recommenders are being trained, evaluated or being used to generate predictions. We would not block a document until they are complete or re-render a document without the user performing an action - but we would show an indicator e.g. saying that new predictions are being calculated or even that they are now available and leave it to the user trigger a re-render if that is desired.

Does that make sense to you?

[munterkalmsteiner]

Yes, that feedback mechanism on training/prediction status would address the UX issue.