Standalone interactive dashboard for antenna data

[tchamberlin]

This, but for antenna data. This is distinct from Django/Alda -- it will stand on its own and interface only with Parquet files on disk.

Separate from this task, more columns will be added to the Parquet tables: Observer, Session Name, Frontend, Backend, etc. These can be used to provide filters via Panel, and data for hover menus later on.

This should not run in a Jupyter notebook -- it should be a Panel "app" that stands independently

[KaseyW31]

I have a Panel server that shows the map, along with selection menus for color map and session name. I'm currently working on filters and widgets for other columns (with fake data)

[KaseyW31]

I have working widgets for other columns, but a problem is that updating any parameter resets the zoom/extents of the plot. The main workaround I've found online is wrapping the view function with a Holoviews DynamicMap, but I don't think it works for this case because the plot itself is a DynamicMap, and DynamicMaps can't be nested. I might skim through the param and panel widget packages more to see if there's another solution

I also had to downgrade Bokeh in my conda env because of Deserialization errors triggered by a specific slider widget. Apparently the issue is somewhat addressed in version 3.2, but conda doesn't seem to know it exists?

[tchamberlin]

Interesting -- I wonder what is different between your notebook and the taxi example?

Which conda channel are you checking? I assume pyviz would have the latest, but not sure

[KaseyW31]

It's working now! I took more inspiration from the taxi code, and it seems like when I split datashade() into its separate operations (rasterize() and shade()), it doesn't generate a DynamicMap anymore, so I can use another one without nesting. Another issue is that for larger parquet files, the server takes a really long time to load

I followed the conda page for Bokeh and tried the bokeh channel, but it didn't update. pyviz doesn't have it either. I also specified bokeh=3.2 -- it failed a bunch of times, then conda found incompatible packages and was spinning for ages. I decided to downgrade for now, which works well enough

[tchamberlin]

It takes a long time to start the server? i.e. to load the parquet file into memory intially? Or changing filters takes a long time?

Regarding the env -- might be worth trying with micromamba instead of conda. It would be wise to make a new env rather than modify your current one. micromamba is available at ~gbosdd/.local/bin/micromamba. We can test out some stuff this afternoon

[KaseyW31]

Unfortunately both starting the server and changing filters take a long time. I'll time different functions to see which are the slowest. Right now, changing one filter also reapplies all the current filters, so I'll work on breaking up the functions that update the plot

I tried micromamba using ~gbosdd/.local/bin/micromamba create -n ant_pos_dashboard, but I was denied permission -- did I use the command properly?

[tchamberlin]

If the filters are slow, I assume that it's re-running some stuff that hopefully doesn't need to be re-run (e.g. projection)

Re micromamba, I forgot that you aren't in the gbosdd group. We'll figure it out later

[KaseyW31]

The current server is pretty snappy after making the following changes:

• Multi-indexed parquet file -- filtering by a single session or observer takes ~0.2s
• Adding x- and y-sampling limits reduces the holoviews rendering time
• Removing projections for now

However, filtering by a list of front/backends or a range of number of scans still takes a few seconds (using df.loc[pd.IndexSlice[...]] or df.query()). I'll be comparing the speed of points.select, normal dataframe masking, and dataframe querying to see if there's one that works faster

[tchamberlin]

I think anything that results in a full-table scan is going to be pretty slow. But a few seconds is tolerable, if it only happens when a given filter is changed. If it happens every time the map is moved/zoomed/etc., then that probably isn't viable

[tchamberlin]

So can you now filter by e.g. a session and an observer and have it still be fast?

[KaseyW31]

Yes, it's similar or faster when filtering by both. It takes about 0.2-0.3s to filter by just observer, but when it's already filtered by one session, the dataset is much smaller, and it takes ~0.003s to apply other filters.

Filtering by just session is a bit interesting -- the first filter takes 1-2s, but future filters only take ~0.001s