search

Ribbit-Network / ribbit-network-dashboard

The web dashboard for the world's largest crowdsourced network of open-source, low-cost, CO2 Gas Detection Sensors.
MIT License
17 stars 16 forks source link

Plan out and possibly prototype what a data pipeline would look like to generate the heatmaps in realtime #76

Open keenanjohnson opened 2 years ago

keenanjohnson commented 2 years ago

We should figure out how to generate emissions heatmaps in real time on our data dashboard.

abhinavtripathy commented 2 years ago

@keenanjohnson real time is actually an interesting challenge, as we look to scale this I wonder if we should consider doing refresh every few seconds first and then pure real time. I wonder what the cost/benefit for the approaches are. I am not sure what you were thinking for this.

nivashini10 commented 2 years ago

I am exploring d3 gallery for heatmaps examples - https://observablehq.com/@d3/gallery. D3.js is a JavaScript library for visualizing data using web standards. The particular example that would be great fit for ribbit data would be Hexbin Map and Bubble Map.

I am currently working to create a mock heat map in D3.js and introduce real time by writing a script for refreshing the data. It would be a great way to start. In JavaScript, there is a timer class that will fetch an update using ajax. I am not familiar with JavaScript and learning as I go, so would love to take any suggestions if someone else has worked on this before!

I also looked into tableau and Matplotlib package in python for heatmap generation and it can be used for real time updates when connected to the server or cloud. Tableau may require license key later on.

fosteman commented 2 years ago

@nivashini10 , did you get much further into d3? I have some practical experience, we can hack something together. Lmk

fosteman commented 2 years ago

Thinking that it'll be great to go ahead with Firebase - offers bunch of services (including serverless functions, firestore DB, and CI pipelines) for free under a certain measure of use - would like to tap into our analytics to figure out current usage stats.