search

Ribbit-Network / ribbit-network-dashboard

The web dashboard for the world's largest crowdsourced network of open-source, low-cost, CO2 Gas Detection Sensors.
MIT License
17 stars 16 forks source link

Let's get it started, again. #95

Closed fosteman closed 2 years ago

fosteman commented 2 years ago

Also houses functions/ that will be separated into API codebase.

TBD:

keenanjohnson commented 2 years ago

@abhinavtripathy mentioned to me on a call today that he was interested in looking over this PR as he has some experience with Firebase.

abhinavtripathy commented 2 years ago

@fosteman Great work! Thank you for the detailed comments on the PR. I was wondering though, if you can explain your motivation to use firebase and how the stack you are using would be better for the ribbit project? I have my thoughts on this but I would love to hear your thoughts and where you are coming from on your approach. cc: @keenanjohnson

fosteman commented 2 years ago

@abhinavtripathy

if you can explain your motivation to use firebase

Free frontend hosting, serverless functions, noSQL db, built-in authentication service, storage and other services to a fair free-tier degree.

  1. Hosting, for example, is much better than a turtle-slow heroku.
  2. Secure serverless functions conform to our needs of collecting data from sensors perfectly.
  3. Authentication service and Firestore that can allow us to authorize frogs, collect usage analytics, create mailing lists, and otherwise have a line of customer support.

and how the stack you are using would be better for the ribbit project ?

I hope it is obvious to anyone who comes across this thread that a python-frontend is a nightmare to maintain and develop in comparison to a react-frontend. And I am not doing anything particularly new: just a Typescripted React project with Material UI and MobX for state management.

  1. This way, we can easily implement a GitBook or design any number of pages.
  2. Create complex interfaces with data-visualizations.
  3. Most importantly, this project separates the code onto API (/functions) and pure frontend deployments, this way no secret tokens or keys are left in the open, and we have a secure API endpoint to serve the needs of the frontend.
abhinavtripathy commented 2 years ago

@fosteman Great points! Thank you for explaining them in detail.

I hope it is obvious to anyone who comes across this thread that a python-frontend is a nightmare to maintain and develop in comparison to a react-frontend.

I absolutely agree with you. python based frontend are harder to scale and maintain. I like react or other alternatives like vue etc. But I do think react would be a good way forward especially considering scale. Today we are using python to get some quick prototyping done but if we want complex visuals and render different pages/graphics, react would definitely be nice!

Free frontend hosting, serverless functions, noSQL db, built-in authentication service, storage and other services to a fair free-tier degree.

While I agree with you that firebase as of "now" is a great tool for hosing/db/auth/storage. I am not sure how that scales as in what we do in the future. As an open source project I think we need to be careful about vendor lock in. I would be inclined to use firebase if we can write classes and abstract out the firebase specific logic into functions so that if we were to switch to a different platform whatever that maybe we are able to do it as easily as possible. I hope that makes sense.

I am also curious to think about other potential platforms we could use. Supabase is one that comes to mind that is an open source alternative to firebase.

cc: @keenanjohnson

fosteman commented 2 years ago

I would be inclined to use firebase if we can write classes and abstract out the firebase specific logic into functions so that if we were to switch to a different platform whatever that maybe we are able to do it as easily as possible.

@abhinavtripathy ,This is absolutely possible. All of the backend is written in standard node.js. And although npm: firebase-functions introduces couple of handling functions - the code is similar to express.js. Migrating it to a different platform will not be an issue.

keenanjohnson commented 2 years ago

Thanks for the good discussion all and most of all, thanks to you for your hard work in prototyping @fosteman ! I've just taken fairly detailed look through the PR.

PR Summary

This PR (correct me if I'm wrong) is a prototype that accomplishes the following:

Product Feedback

Technical Feedback

I'm a bit fan of breaking PRs down into small chunks in software projects, as it helps us not conflate pros /cons of different options together.

I would prefer to tackle each of the items I listed above separately as I don't believe they all have to happen together.

React Front End

For the Front end, sounds like the two of you agree that a react front end would serve us better in the long term. That's great and I'm not a excellent web dev, so I'll defer to your judgments there. Is it possible to build the other features we have in the current web front end into this prototype (things like plots, time horizon queries, CSV exports, etc.)?

Switches Database to Firebase

In the latest discussion in #81 sounds like the consensus is not to use firebase. I haven't heard a convincing argument about why Firebase is superior other than it is fast to prototype, but perhaps I am just missing something.

Firebase API

Reading through the code, it does seem like Firebase is a fast solution to build out a proper REST API. Perhaps that is relevant in the pricing discussion in #81 for someone to bring up? It wasn't obvious to me how the pricing structure works for that though. Does anyone have any experience there?

Security Changes

I believe these can be made with any tech stack so let's make progress on that in isolation.

Keenan's Proposed Route Forward

How does that sound @abhinavtripathy and @fosteman ? Any thoughts or questions?

fosteman commented 2 years ago

Uses the firebase API syntax for the front end queries instead of the past InfluxDB query syntax

Not the case. It is still hooked up to InfluxDB through an API provided by serverless functions deployed separately - our new backend.

fosteman commented 2 years ago

Is it possible to build the other features we have in the current web front end into this prototype (things like plots, time horizon queries, CSV exports, etc.)?

Absolutely. This is an MVP to show the potential and flexibility of the tech stack

fosteman commented 2 years ago

I would prefer to tackle each of the items I listed above separately as I don't believe they all have to happen together.

Open a separate issue and start work on a react front end that has all of our existing features and allows you to toggle on the "heatmap" mode.

Agreed. Let's create a list of github issues to roadmap the features needed and get on it! @keenanjohnson

fosteman commented 2 years ago

In the latest discussion in https://github.com/Ribbit-Network/ribbit-network-dashboard/issues/81 sounds like the consensus is not to use firebase. I haven't heard a convincing argument about why Firebase is superior other than it is fast to prototype, but perhaps I am just missing something.

I'd like to learn more about the benefits of Firebase in the REST API and pricing model around that before making a call there.

It does not "switch" to Firestore. I've used Firestore for SSO Auth purposes on the frog PR, but this PR is devoid of any Firestore influence.

So we're still considering the use of GridDB... This PR is ambivalent to that consideration @keenanjohnson

fosteman commented 2 years ago

Reading through the code, it does seem like Firebase is a fast solution to build out a proper REST API. Perhaps that is relevant in the pricing discussion in https://github.com/Ribbit-Network/ribbit-network-dashboard/issues/81 for someone to bring up? It wasn't obvious to me how the pricing structure works for that though. Does anyone have any experience there?

I have several live projects running on this tech stack. Firebase serverless functions is the right approach here. But if anyone has other thoughts - shoot

Open a separate issue for the security fix

This problem is just the problem of not having a backend - you're storing security tokens to access InfluxDB right in frontend - no bueno. Having a secure backend solves the issue. Hence the Firebase serverless. @keenanjohnson

fosteman commented 2 years ago

Moreover, about the pricing, I'd be happy to hook it up to my credit card. Open source also means monetary contribution. This applies to the VM we will spin up to run DB, I'd be happy to contribute a dime once we need to scale it.

keenanjohnson commented 2 years ago

Again, thank you truly for all your hard work @fosteman !

So is the main point of using Firebase serverless here to fix the security issue with storing the token then or are there other benefits? Sorry for the slow understanding here as I am not an expert at this stuff?

Appreciate your willingness to pay with your personal card, but as Ribbit is a formal non-profit, it's important for our legal / tax situation that all money goes through the official ribbit channels for cloud purchasing, etc.

fosteman commented 2 years ago

So is the main point of using Firebase serverless here to fix the security issue

Precisely, it's the correct way of deploying these things. You need backend of some sort. Serverless is the way to go.

fosteman commented 2 years ago

Appreciate your willingness to pay with your personal card, but as Ribbit is a formal non-profit, it's important for our legal / tax situation that all money goes through the official ribbit channels for cloud purchasing, etc.

I'm onboard with whatever.

keenanjohnson commented 2 years ago

I see. I think I missed the main point of the conversion to Firebase then. Thank you for the patience in explaining it to me :) .

Is the "classical" way this is solved by implementing a REST API in between the InfluxDB system and the Front end like the GO API we have almost complete, but not yet deployed?

fosteman commented 2 years ago

Is the "classical" way this is solved by implementing a REST API in between the InfluxDB system and the Front end like the GO API we have almost complete, but not yet deployed?

I suppose, yes, you'll just be fetching another backend with the same REST.

keenanjohnson commented 2 years ago

Got it. So given that we'll need a REST API likely for other purpose (scientific data access) what the pros / cons of each approach in your experience?

fosteman commented 2 years ago

None, the deployment platform just runs the code, be it node.js or Go.

And honestly, it does not matter whether it is Go or javascript. Javascript can do everything Go can.

Which one is easier to develop ? Either, again. So it comes down to who will be working on the codebase. I'm here with my js tech- stack if you were to ask @keenanjohnson

keenanjohnson commented 2 years ago

Ok @fosteman thank you for explaining everything. I would like to break this down into two main features here. I have created two Github issues linked below that document each for the future. I would like to propose that we tackle them separately in two different Pull Requests @fosteman so that we avoid conflating the two issues.

Convert Site to React

There is no contention on the path forward here it seems. @fosteman your current work on the front end is awesome, though I don't think it's mergeable until we have feature parity with the existing front end. I believe I have documented all the needed features in #96. It's possible to replicate the current feature set I believe by still making the direct influx queries from Influx, so I don't think this depends on the security enhancement.

Hiding Database Keys

I've created a new issue for discussing this specifically in isolation. I'll need to read a bit more about the pros/cons of Firebase serverless before I had an informed opinion. #97

Path Forward

By deconflicting the two issues, you can jump right into #96 I believe @fosteman and we can continue discussing the second issue after I do some more research in #97 and do that as a later PR.

How does that sound?

todbott commented 2 years ago

Nice thread here! I've never worked with Firebase, but I love the idea of serverless. I once wrote a 3,000 line backend in Python (using the Flask framework) and deployed it on the GCP app engine, what a nightmare!

This all looks good to me, nice job @fosteman

fosteman commented 2 years ago

To hell with it

keenanjohnson commented 2 years ago

Hello, @fosteman it seems as if you are frustrated, and I'm happy to discuss alternate paths forward vs the one I proposed above if you feel an alternative is better.

However, there is no reason to curse and I find that behavior quite rude and insulting to both myself and the community.

fosteman commented 2 years ago

I just looked through the feature requests \ issues you created, @keenanjohnson . My concern is that you've taken screenshots of the status quo and hence the requirements. My concern is that those requirements have no user analytics backing, nor any other backing except for out "wants" and "nice to haves", and I'm not okay with that. We need to approach feature development from the standpoint of actual feedback.

Now, I understand that passing through this PR is a regression, however, I will allow this to become a rework of legacy systems. I can only allow this to become a "re-think" of our user experience.

For now, as I proposed, it's a simple map. And I'd like to get on and expose more data. But it can't be run with such requirements.

Everybody here in OSD is contributing their time and passion toward the mission, but no one wants even one more ticket they need to close - we do this full time at our jobs. We do OSD because of the freedom it gives, and because of the real impact it may impart some day. We fall into flow and develop things because it feels right, not because there are deadlines or tickets. I've done some work, and frankly, I don't feel like reworking the code I put my flow time in. I feel like continuing to harvest the flow and keep building momentum instead of stalling and controlling the outputs.

I would like to propose to rework the issues into less tangible - not output-driven, yet outcome-driven feature requests. And I'd like to have feedback on them. I'd like to summon feedback from every possible angle, to stir the interest in the crowd and our core team.

fosteman commented 2 years ago

And frankly, I do not agree with the structure of the issues themselves.

If you were to think about an outcome instead of an output, you'd come to something like:

  1. Deploy an MVP with a modern stack that's based on status quo.
  2. Begin collecting feedback and designing a better UX based on the set of features in the legacy system.
  3. Continue to develop the app, introduce better UX for the newly designed features.
keenanjohnson commented 2 years ago

Hey @fosteman a few opinions from me:

Issue Structure / Separation

I think it's important to separate implementation from issue structure. So I think what you are saying, is that in practice, it might make more sense to implement both the React conversion and the DB key access task at the same time, but they do address different problems, so from a project structure and ability for future people to look back at the history of the project, it makes sense it describes the problems separately. Then we can link both issues to a single pull request if they are solved at once, but in practice, they might be solvable separately.

Web Page Features

I agree with you that the process you described is correct and I have done that which is how we have arrived at the current feature set. Myself personally use those features and I know that other sensor community owners know them. I'm also open to refining the UI for the features, but it's my understanding that we do need to have the features in some way.

If you would prefer to iterate towards those or go back and do customer feedback, then that's fine, but we can't move the old code into a deprecated folder yet as that will break the deployment for the live site now.