search

Ribbit-Network / ribbit-network-frog-software

The software for the Ribbit Network Frog Sensor
https://www.ribbitnetwork.org/
MIT License
9 stars 7 forks source link

Build Open-Fleet Provisioning Page using the Golioth.io Fleet APIs #6

Open keenanjohnson opened 2 years ago

keenanjohnson commented 2 years ago

One of the promising device management solutions for a microcontroller based Frog Sensor is a platform called Golioth.

See the related issue on prototyping a Golioth ESP32 Device: https://github.com/Ribbit-Network/ribbit-network-frog-sensor/issues/126

One of the key question marks at this point is how to implement the "open-fleet" concept with Balena.

Balena Open Fleet

Ribbit Network devices are currently managed as part of a Balena Open Fleet.

An OpenFleet is a project where members can join by simply adding their devices (no balenaCloud account required for users just wanting to add a device). The owner maintains the fleet as an open, non-commercial project. Open Fleets have no device limit, encouraging owners to try and get more users to support and grow their projects. Ribbit Network qualifies being an open-source, not-for-profit org.

A few key Ribbit Staff maintain access to the Balena backend and coordinate fleet-wide updates, but anyone can view the source code and add their own device to the fleet if they would like.

Open Fleet in Golioth

Golioth currently doesn't have any capability for an open fleet. However, I have spoken to the team at Golitoh and they are excited about the idea and supporting Ribbit Network by offering a possibly reduced price.

However, in order to join the open-fleet in Golioth, we will need to build some software that calls into the Golioth APIs.

We should start by reviewing the Golioth APIs, and ensure that all the APIs are available to build a site that would allow people building their own Frog Sensors to join the Golioth Open-Fleet.

After that we should be empowered to build out that functionality!

beriberikix commented 2 years ago

👋 hello from Golioth! Let us know how we can help with the prototype/concept!

keenanjohnson commented 1 year ago

From a high level this is what we need someone to build a prototype to ensure that the Golioth API contains all the items needed to accomplish the following so that we can re-risk using an ESP32 device with Golioth as an open solution for our open-source network.

Brief Description We need to create a prototype website / web service that allows anyone in the open-source community to provision and add an ESP32 device to a Golioth fleet.

High-Level Flow

beriberikix commented 1 year ago

Great! That high-level flow makes sense and should be do-able 🙌 Some data points:

Before building the RNOW you can test this flow with the goliothctl commandline tool.

Now on the firmware side there will need to be more discussion. If the prebuilt binary will be a "golden" image, ie same for everyone, than one way to get credentials on the device is via the terminal (see the example.) The RNOW could also build a custom image that includes Golioth + Wi-Fi- credentials, but that will add non-trivial complexity.

Two other options that are also worth exploring:

  1. Espressif has a Unified Provisioning library, which includes Wi-Fi or BLE provisioning and even mobile app SDKs (deps on hardware.) We haven't integrated it yet into our SDK but it should be possible in the future.
  2. Certain Espressif devices, like the ESP32-S2 & ESP32-S3 have native usb and can be programmed via drag and drop (they show up like USB drives.) We could in theory use that as a way to get credentials on the device without opening a terminal or need a mobile app. Note: I'd personally recommend using the S3 over the classic ESP32. It is more capable for not much more money and Espressif is recommending it for new designs.
todbott commented 1 year ago

Hello @keenanjohnson , I just confirmed everything that @beriberikix wrote above! Using the API, I was able to add a new ESP32 to my Golioth dashboard, then begin receiving data from it.

I'll make a little mockup web portal in the next week or so, so we can see it in action. Very simple and cool!

keenanjohnson commented 1 year ago

Awesome! That is great @todbott ! Did you look into any of the challenges @beriberikix mentioned with flashing the prebuilt binary onto the device?

Perhaps it might be helpful to create a process flow diagram for the specific API calls needed + local command line commands and user actions to provision, flash, and join a device to a golioth fleet as a next step here @todbott ?

beriberikix commented 1 year ago

Charting out sounds like the right next step! Ya, it'll come down to which ESP32(s) you want to support. Also, do you expect anyone making their own frog sensor to be comfortable with the command line?

I can come up with even more ideas but I'll hold off until we answer some more questions 😊

fosteman commented 1 year ago

Also, do you expect anyone making their own frog sensor to be comfortable with the command line?

I'm thinking that it narrows down our potential user base substantially. Having a user-friendly interface and installation instructions and\or a ready plug-and-play product is crucial.

todbott commented 1 year ago

Thank you, here is a workflow I used late last week to add a new ESP32 to my Golioth.io dashboard and begin receiving data from it.

This workflow assumes that Ribbit Network already has a Golioth.io account and project. I used my own account when thinking of this onboarding flow--my project is named 'ribbit-practice', so that is the project name I'll use in the API calls below.

Getting the user a hardware ID

So, the user starts out with an ESP32 and a USB cable. They go to a special web portal, and click on "Join the fleet".

When they click on "Join the fleet", we can send a POST request to the Golioth API, this endpoint and this JSON:

/v1/projects/ribbit-practice/devices
{
  "name": "an arbitrary name for the new fleet member",
  "hardwareIds": [
    "a unique and random string"
  ]
}

As a result, we will receive a JSON response with a field called 'id'. A space for the new fleet member (and an associated ID) has now been created in our Golioth.io dashboard.

{
  "data": {
    "id": "6311a4752aa9e6655dabf779",
    ...
}

Getting the user a device identity and PSK

In order to send data to our Golioth.io dashboard (and also be available to receive any future firmware updates), users need to input an identity and PSK (pre-shared key) into their device, and the same identity and PSK have to be registered in our project. The identity and PSK can be anything, really. The necessary thing is to register them in our Golioth.io project, essentially letting the system know "You will be receiving data and doing firmware updates on a device with this identity and PSK in the future." So, another POST request to the API, with which we add an identity (string) and pre-shared key (string) for the new fleet member we created above:

/v1/projects/ribbit-practice/credentials
{
  "deviceId": "6311a4752aa9e6655dabf779",
  "type": "PRE_SHARED_KEY",
  "identity": "some kind of unique string",
  "preSharedKey": "some kind of unique string"
}

The identity (which we need to put in the code) now becomes the 'identity' in the JSON, with the suffix '@ribbit-practice' (the project name). The pre-shared key also needs to be put in the code.

From there? The hacky way forward:

Of course, a person could just copy-paste this ID and PSK into their code (along with their wi-fi credentials, like I did here:

    wifi_init("IODATA-cdbc80-2G", "8186895522346");
    wifi_wait_for_connected();
    golioth_client_t client =
            golioth_client_create("id@ribbit-practice", "psk");
    assert(client);

I was able to get my ESP32 (newly added to the fleet) to begin streaming data to the Golioth.io project dashboard, in this way.

Flashing is the real bottleneck

In this way, we could have a flow like this:

The bottleneck here (the part that will make it very hard for users) is the need to download the ESP-IDF and install it. I thought perhaps we could just have a VM in the cloud somewhere that could take as input 1) the identity and PSK and 2) the user's wifi credentials, then build the necessary .bin file (using idf.py build), and present them with that .bin file... however, as far as I know flashing data to an ESP32 requires the ESP-IDF. So, it looks like the user will need that tool eventually, regardless of what they do.

beriberikix commented 1 year ago

Great writeup! My recommendation is to have a strongly opinionated happy path but still make alternatives possible. This is especially true when designing technical products for non-technical users. Trying to make something that is easy to use with a consumer-level setup experience AND supports multiple hardware variations is going to be tough.

Some more notes to ponder:

So my recommendation would be:

keenanjohnson commented 1 year ago

This seems reasonable to me as a recommendation @beriberikix !

@todbott does that make sense to you? Would you have any time or interest in prototyping the web-based setup flow similiar to esp-web-tools to compliment the work you have already done? Completing this is the main blocker is deploying more Frogs right now as Raspberry Pis are nowhere to be found :(

keenanjohnson commented 1 year ago

Seems like @todbott is not able to continue on this, so this issue is open for anyone that has interest to tackle!

todbott commented 1 year ago

Thank you @keenanjohnson , yes, sorry for the long silence! An old client contacted me recently, and now I'm in the middle of building a whole React.js billing dashboard for them (to be deployed for the whole APAC region), so my available time has decreased dramatically.

Also, in preparing for this project, I accidentally bought an ESP32 that (it turns out) has a reputation for being crazy, and is not recommended for use anymore (a WROOM-32C, made by a company callled "Wave").

In the work I have done on this issue, however, I've learned so much about Golioth, so I'd love to chime in occasionally here to help someone along.

keenanjohnson commented 1 year ago

One big decision we will need to make is whether we are going to store user profiles for people building sensors? We haven't historically done that. Here are my thoughts

Pros:

Cons

The pros would be nice imo, though the security / data stewardship challenges trend me to believe that we should try to use something out of the box to limit our surface area for data leakage, etc. What do you all think?

beriberikix commented 1 year ago

If you decide to support and/or require login, I'd advise outsourcing it to a trusted third party like Auth0 and Firebase (there are many.) They can greatly reduce the risk of leaks, PII concerns, GDPR, etc.

In general it might be a good idea, not just for service communication but also managing abuse. Anonymous API users could be a pretty scary vector.

keenanjohnson commented 1 year ago

Agreed. So I think the list of subtasks that need doing are roughly this:

@abhinavtripathy mentioned in Discord that he would start looking into this tomorrow. Let me know what support you need.

abhinavtripathy commented 1 year ago

@keenanjohnson my thoughts so far:

Asks from you @keenanjohnson :

abhinavtripathy commented 1 year ago

in terms of an ECD, I am still working towards that. I will have a design doc shortly as well.

damz commented 1 year ago

Firebase sounds reasonable to me, as we are likely going to rely on other Google services (i.e. Pubsub and BigQuery)... but it is still a lot of reinventing the wheel. If we go with something like Django for the backend we get user authentication for free.

(Additionally, if we go with Django we don't have to go through the hassle of building separate backend and frontend applications. At the cost of sticking with an old-style backend driven frontend... but do we really need anything else?)

abhinavtripathy commented 1 year ago

Fair point, we can ahead with firebase.

but it is still a lot of reinventing the wheel.

not sure I understood what you mean by that, could you please explain more.

If we go with something like Django for the backend we get user authentication for free.

True, though I am not a huge fan of Django simple because it old style. Is the auth cost a big consideration for this? I propose React on the client side and we can go ahead with firebase functions or some other serverless options for the backend.

mlbenthall commented 1 year ago

It seems like it would be really nice to:

Is there any identifier that the sensor/ESP32 can instrospect about itself, that we could use as its key, or construct its key from? We'd need it to be something "unique". Also something that the user can discover and type in when registering.

beriberikix commented 1 year ago

Yup, every ESP32 (and just about every Wi-Fi device) has a unique ID based off it's base MAC address: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/misc_system_api.html#mac-address

For future devices that are not wi-fi, like cellular, also have a unique ID like IMEI.

keenanjohnson commented 1 year ago

Great points @mlbenthall and @beriberikix !

The software keys are modifiedable at runtime. You can see the instructions for how to set them here.

I think having the devices auto-provision could be a nice future goal, but I still think having the registration page would be valuable so that we can collect a contact email associated with the sensor to do future things like notify them if the sensor goes offline, etc.