Proof of Concept: read locations, parameters and timeseries from databricks

d2hydro / hydrodashboards

Open Source Dashboards for hydro data

MIT License

2 stars 1 forks source link

Open d2hydro opened 1 year ago

d2hydro commented 1 year ago

Ultimate goal is to be able to read from Azure as a source for the WIK hydrodashboards app.

With this issue we purely test performance with a basic POC bokeh dashboard

Actions we take:

[x] Read locations via pd DataFrame into bokeh Select object
[x] Read parameters via pd DataFrame into bokeh Select object
[x] Read timeseries via pd DataFrame into bokeh Column DataSource shown in a Bokeh figure
[x] User can update data in Bokeh figure when selecting a location/parameter

d2hydro commented 1 year ago

Required improvements on AAM side:

[ ] add a list of available parameters per object in silver_v2.dim_measurement_object
[ ] Include grouping of objects with their time-series (equivalent to FEWS filters)

Nice to haves from AAM side

[ ] SQL statement for getting the header of a databricks table, as we won't be able to construct it if the List[Row] result gives an empty list
[ ] sampling/filtering/shading of large timeseries (> 100000 values) for dashboarding (so we won't need to do that on the bokeh-side