Develop a proof-of-concept prototype for a web-based graphical interface

[niksirbi]

Feature idea: interactive visualisations in Jupyter

During discussions with @Di-Wang-AIND and Sharmishtaa Seshamani from the Allen Institute for Neural Dynamics we determined that there is a lot of interest in a web-based graphical interface form movement, specifically one that would allow for interactive data visualisations within Jupyter Notebooks. This could be very useful, since lots of researchers are comfortable with working in notebooks, especially during data exploration stages (where interactive visualisation is most needed). It would also facilitate cloud-based workflows.

Context: web-based vs desktop (napari) frontends

For context, we are in parallel developing a desktop graphical interface via a napari plugin (see this meta-issue and ongoing work-in-progress). At this early stage, we think it makes sense to explore both options (napari-based and web-based) and evaluate their respective strengths and limitations. They should be independently developed but strive to present a consistent interface (as far as possible). Long-term we may decided to stay with only one frontend, or keep both.

Example interactive visualisation workflow

This is an envisioned hypothetical workflow through a graphical frontend from the user's perspective:

• Select file containing pose predictions (probably via a file browser widget), the pose estimation software it was created with and the fps
• Optionally select the corresponding video file to show as underlay (could be another file browser widget). There should also be the option of using a static image as an underlay (may be a still frame or a schematic of the arena, for example)
• Explore the loaded data - the keypoints overlaid on the video/frame - with interactive widgets to adjust the shown individuals, keypoints, etc. (including option to plot a "tail" for a certain time window back in time).
• Generate and save video/clips or plots (for figures, presentations, reports). This part is less fleshed out in my mind, because so far I've been focusing on implementing the previous bits.

The end-product may look something like the mockup below:

The above mockup is not to be taken literally, it's still on a conceptual level. The only part of it that's underway on the napari side is the file-loader widget:

Potential frameworks to use

So far we've brought up fastplotlib and plotly. That said, @Di-Wang-AIND has identified some potential blockers on the fastplotlib side, see https://github.com/fastplotlib/fastplotlib/issues/376 and https://github.com/fastplotlib/fastplotlib/issues/398. Alternative frameworks may still be considered.

How to move forward

I think it would be best for this exploration to start on a separate branch, ideally leading to some demos. The first implementation need not be "clean", it should rather serve as a proof-of-concept. After evaluating that, we could move onto slowly merging the functionality into the main branch, ideally in the form of atomic PRs (each complete with tests).

@Di-Wang-AIND do let me know if you want to take this on within the next two months.

[niksirbi]

BTW fastplotlib works in jupyter, but the exact same code can also run as a desktop app (this finally behaves properly after https://github.com/fastplotlib/fastplotlib/pull/431 ).

Thanks @kushalkolar !

In light of this, it's worth reconsidering our graphical framework-of-choice: if fastplotlib can work with various graphical backends (jupyter + desktop) we might as well reconsider and focus our efforts on fastplotlib. Building the prototype will be crucial in informing our decision.