Sync applets across dashboards with master-running AppletDB class

[margieb]

ARTIQ Feature Request: Applets run with the master and are universal to different dashboards

Problem: Currently, applets are saved separately for different dashboards

We are using the headless repository setup and running the master on one machine and several (usually qty 5-10) dashboards sending experiments and populating datasets to/from the master. We would like to have the applets shared among dashboards automatically at their creation in a similar way to the datasets. Currently, when we create an applet in one dashboard we have to copy/paste it into the others for them to have the same. When we change an applet we have to apply that change to each dashboard. When several different experimenters are working on different experiments across several machines, it becomes very likely that various test versions of applets or old versions will exist in a dashboard and prevent robust data visualization. We are aware that some people use the CCB to broadcast applets to all dashboards at the runtime of an experiment but this is not the most efficient plan for us since we would have to write the CCB into every experiment that we want a different applet for and/or run it separately from the actual experiment to get the applet in every dashboard. This is especially limiting in cases where we don't know exactly how we want to visualize the data and may be making changes to the applets in parallel with changes to the experiments.

Proposed Solution: Add an AppletDB class in addition to ExperimentDB (in experiments.py) and DatasetDB (in databases.py) to run with the master.

This would make applets consistent with how experiments and datasets work in the framework of distributed dashboards with a single master (ultimately more modular and scalable). This would enable us to have the applet list automatically populated, sync changes to applets among dashboards in the same manner as we do for experiments and datasets, and (with the master saving an applet_db.pyon) git-track applets in the same way as datasets. This would have the added benefits of dealing with potential inconsistencies when one user edits a CCB-generated applet on only one dashboard and, with careful git tracking, recovering old applet code overwritten by the CCB. The enable/disable checkbox could be independent of what the master knows about the applets and save as part of each local dashboard's config files.

RFC

Does this conflict with how anyone else currently uses things? Are there any alternative solutions that are just as or more consistent and efficient? Are there any fundamental challenges to this implementation that we are misunderstanding? Any modifications to this idea itself? Thanks!

[sbourdeauducq]

Might be reasonably straightforward to get pyqt/pyqtgraph output into a virtual framebuffer, and forward the framebuffer (possibly with VNC/rdesktop-style compression), and mouse and keyboard events between dashboard and master. Qt used to have something called qvfb that we can look into.

But it would break, at least:

• any applets written for toolkits other than Qt.
• pyqtgraph functionality that opens additional windows, such as the "export to PNG" feature.
• most likely any OpenGL acceleration.

[margieb]

@sbourdeauducq What I was suggesting here was just to have the applet data (name, command) saved in an applet database which is managed by the master so that the list of applets can update automatically in each connected dashboard in the same way the datasets do. It sounds like your solution here is to actually run pyqt from the master? Do you see an added benefit to going that far? I am not 100% sure the mechanics of how the datasets update but I was imagining the list of applets could be updated in a similar way with an AppletDB class and database. This shouldn't change how the applets run within individual dashboards, so it should avoid breaking the things that you mention here.