Designing more polished UI with Streamlit

[nellatuulikki]

1. User can choose the model
2. Button for compiling
3. Button for installing
4. Visualization of the pipeline (first 6 steps of the life cycle to the home page)
5. Map element, similar to https://streamlit-demo-uber-nyc-pickups-streamlit-app-456wus.streamlit.app/ but this can be animated

[doyu]

You can control color of Streamlit via https://docs.streamlit.io/library/advanced-features/theming

[doyu]

https://youtu.be/wyWmWaXapmI?t=4960

[doyu]

Real time drawing when IoT devices are moving

https://github.com/ash2shukla/streamlit-stream https://blog.streamlit.io/how-to-build-a-real-time-live-dashboard-with-streamlit/

[FexbYk23]

For number 4 I found this: https://github.com/Mohamed-512/Extra-Streamlit-Components

The TabBar and StepperBar both seem like decent options, but both have their flaws. The TabBar has no logic for preventing the user from choosing tabs out of order and the StepperBar has no option to set the default value.

[doyu]

I suppose that choosing order may not be so important here. Basically the flow itself seems to have been already kinda self-explanative in the order of the current side bar menus items. IIUC, Roberto wanted that the frontpage should present which items are choosen in each phase? Or do you want to replace the side bar with pipeline? @robertmora confirm here? Any visual example in your mind?

IMHO, at worst, this could be satisfied even only with each page puttuing what's choosen on the top of its page, if available visual pipelining options are somewhat too overengineering (i.e. big effort on visualization). Although we, of course, want decent UI but we have originally choosen Steramlit for the trade-off point between effort and easiness. IOW, if we really want an ultimate fancy UI, we should have gone with React or something from the biginning. But that's not our target, at least, for now. We really want to tell our story to investors with this app, what we wil achieve for customers ;)

To decide which solution / UI to take, @robertmora should provide somewhat more info here.

[doyu]

For 5, an example of simple real time updating:

Or https://towardsdatascience.com/simple-plotly-tutorials-868bd0890b8b

[robertmora]

I can discuss with the team about this tomorrow as well. Anyway, my idea was to still keep the sidebar with pipeline items and in the front/home page placing this sort of step bar that provides an overview of the current operations/tasks if a new pipeline process or an pipeline step updated is being executed.

But, at this stage, if it requires too much time, let's forget about it and let's try to focus on low hanging fruit activities.

In general, the platform should always be able to provide an overview about all the devices connected in the TinyMLaaS tab. Then, as soon as the user wants to add a new device and/or install a new version of the ML model in a specific device and/or compile with a different quantization and/or select a different data source for device, etc., it should be able to start a new process (can't come up with a proper name for it) that allows to do perform the whole pipeline or a single task. So, thinking at the big picture, I do see the platform and each sidebar item (a complete version of if, so not something that should be implemented for this sprint) providing the following capabilities:

TinyMLaaS

• Connected devices: This tab provides a list of all the devices that are currently connected to the platform. It could also display basic information about each device, such as its name, status, and location.
• Device map: This tab displays a map that shows the locations of all the connected devices. This could be useful for users who want to quickly locate a device and see its status.

Devices

• Add new device: This tab allows users to add a new device to the platform. Users would need to provide some basic information about the device, such as its name, type, and location.
• Update device: his tab allows users to update the information for a device that is already connected to the platform. Users would need to select the device they want to update and then provide the new information.
• Select device for ML model installation/update: this tab allows users to update the information for a device that is already connected to the platform. Users would need to select the device they want to update and then provide the new information.

Data

• Data source selection: This tab allows users to select a data source for the ML model. The available data sources will depend on the device that was selected in the previous step. Users might be able to choose from options such as sensors, cameras, or pre-recorded datasets.

Model

• Model selection: This tab allows users to select an ML model for the selected device and data source. Users might be able to choose from a list of pre-trained models or upload their own custom models.
• Model versions: This tab allows users to select a specific version of the selected ML model. Users might be able to choose from different versions that have been trained on different datasets or with different parameters.
• Quantization selection: This tab allows users to choose the level of quantization that they want to apply to the ML model. Quantization can help to reduce the size of the model and make it more efficient for deployment on a device.
• Other ML model parameters (if any): This tab allows users to configure any additional parameters for the ML model, such as hyperparameters or preprocessing steps. These parameters will depend on the specific ML model that was selected.
• Compatibility check with data source and device: This tab checks whether the selected ML model is compatible with the selected data source and device. If there are any compatibility issues, users will need to adjust their selections before proceeding.

Training

• Training settings: This tab allows users to configure the settings for the ML model training process. Users might be able to choose from options such as the number of epochs, batch size, or learning rate.
• Training process: This tab displays information about the training process for the selected ML model. Users might be able to view metrics such as accuracy, loss, or validation error.

Compiling

• Compilation settings: This tab allows users to configure the settings for compiling the ML model for deployment on the device. Users might be able to choose from options such as the target device architecture, optimization level, or input/output formats.
• Compilation process: This tab displays information about the compilation process for the selected ML model. Users might be able to view metrics such as compilation time, model size, etc.
• Model validation: This tab validates the model after compilation. Users might be able to see metrics such as accuracy, latency, or power consumption. They might also be able to compare the performance of different models or configurations.
• Model packaging: This tab packages the compiled model in a format that can be installed on the target device. Users might be able to choose from different packaging formats or specify custom packaging options.
• Packaging status: This tab displays the status of the packaging process. Users might be able to see progress bars, error messages, or other information about the packaging. This tab can be useful for monitoring the packaging process and resolving any issues that might arise.

Installing

• Installation settings: It lists a summary of all the previous selected steps.
• Installation status: This tab displays the status of the installation process. Users might be able to see progress bars, error messages, or other information about the installation. This tab can be useful for monitoring the installation process and resolving any issues that might arise.

Observing

• Device output: It displays the output of the ML model running on the selected device. Users might be able to see real-time data coming from sensors or other sources, as well as the predictions made by the ML model.
• Data visualization: It allows users to visualize the data being generated by the device and the ML model. Users might be able to see graphs, charts, or other visualizations of the data over time.
• Model performance: It displays information about the performance of the ML model running on the device. Users might be able to see metrics such as accuracy, latency, or power consumption. They might also be able to compare the performance of different models or configurations.
• Device control: It allows users to control the device and the ML model running on it. Users might be able to start or stop data collection, adjust the model parameters, or send commands to the device. This tab can be useful for debugging or fine-tuning the ML model in real-time.

[doyu]

My idea was to still keep the sidebar with pipeline items and in the front/home page placing this sort of step bar that provides an overview of the current operations/tasks if a new pipeline process or an pipeline step updated is being executed.

One way to do this with the current side bar may be using session state[1]. With keeping the phase transition history in this session state, each side bar item (each phase) could have been marked checked / unchecked icon at the top of item line, which indicates done or to-be-done state respectively. Even some pull down menu under each item would show what options were choosen so far in those phase too. If we successfuly put all of such info in side bar, which is always visibule, we might not need visual pipeline on the frontpage? Then what's on the front page???? I have no idea, right now, though ;) ..... Any opinion?

[1] https://docs.streamlit.io/library/api-reference/session-state

[robertmora]

The front page could easily show this:

– Connected devices: This tab provides a list of all the devices that are currently connected to the platform. It could also display basic information about each device, such as its name, status, and location. – Device map: This tab displays a map that shows the locations of all the connected devices. This could be useful for users who want to quickly locate a device and see its status.

(The idea would be that our TinyMLaaS customer must manage a relative big number of devices.)

[doyu]

– Connected devices: This tab provides a list of all the devices that are currently connected to the platform. It could also display basic information about each device, such as its name, status, and location. – Device map: This tab displays a map that shows the locations of all the connected devices. This could be useful for users who want to quickly locate a device and see its status.

A Device map followed by (connected) Device table below would look cooler. Device map may show green dots for connected device while red dots for not connected ones. If you click a red / green dot, the detail info should be listed on the top of table below, for example :D All of this feasibility depends on implementation effort, though.

Logically the above info belongs to Device page in principle, but it would look cool if the front page shows the above info on live :) Actual Device page could show its device registration form and a list of registered devices in a table simply, then.

[robertmora]

Yes yes, that was exactly my idea ;) The front page should provide an overview about the devices registered into the network (online and offline), the ones reporting errors, the ones reporting data drift risks, the ones having low-battery, the ones having poor connectivity, etc. Differently, the Device page would be the interface for dealing with the device registration, update, etc, as wrote above.

[doyu]

I checked st.map(). It doesn't have much complicated features. It doesn't seem to have red/green entities. We could go with st.map() now and if we need more sophistocated feature, we could consider later.

[robertmora]

Text for tabs description:

TinyMLaaS displays a list of all the devices that are connected to the platform. Users can view the locations of these devices on a map. It also allows users to manage and monitor their devices, including adding new devices, removing devices, and checking the status of each device.

Device displays a list of all the devices that are connected to the platform. Users can add new devices to the platform and select an existing device for installing or updating an ML model. It also enables users to manage their devices, including viewing device information, modifying device settings, and monitoring device status.

Data allows users to select the data source that corresponds to the device selected in the previous pipeline step (i.e., Device). Users can choose from different data sources, including sensors, cameras, or other input devices. It also ensures that the selected data source is compatible with the device and the ML model.

Model enables users to select the ML model they want to install on the device. Users can choose from different pre-built models or upload their own custom models. Users can also select the quantization level and other parameters of the ML model. It also ensures that the selected ML model is compatible with the selected data source and device.

Training allows users to train the selected ML model. Users can choose from different training algorithms, hyperparameters, and other settings. It also ensures that the ML model is trained properly and can achieve the desired accuracy and performance.

Compiling enables users to compile the selected ML model for the target device. Users can select the compilation settings, including the target architecture, optimization level, and other options. It also ensures that the compiled model is optimized for the target device and can run efficiently.

Installing allows users to install the compiled ML model on the target device. Users can select the device and the compiled model, and monitor the installation process. It also ensures that the compiled model is installed correctly on the target device and can run as expected.

Observing enables users to observe the selected device in action, while using the installed ML model and processing the selected data input. Users can view the output of the ML model, including predictions, classifications, or other results. It also allows users to monitor the performance of the ML model and the device in real-time.

[doyu]

TinyMLaaS displays a list of all the devices that are connected to the platform. Users can view the locations of these devices on a map. It also allows users to manage and monitor their devices, including adding new devices, removing devices, and checking the status of each device.

We consider this front page, TinyMLaaS, as dash board of WebApp. This should show the overall information about registered devices and some statistical data (e.g. # of devices, # of observatios, total # of detected human and so on). But this page shouldn't provide any control feature itself. control feature should be done in each sub pages. Adding new device should be done in Device subpage.

Device displays a list of all the devices that are connected to the platform. Users can add new devices to the platform and select an existing device for installing or updating an ML model. It also enables users to manage their devices, including viewing device information, modifying device settings, and monitoring device status.

As we discussed above, basically subpages are flow from top to bottom. We still don't associate a device with other entities yet here (e.g. Model, Data, Installer) yet. That kind of association should be done later in Installaing.

[doyu]

Data allows users to select the data source that corresponds to the device selected in the previous pipeline step (i.e., Device). Users can choose from different data sources, including sensors, cameras, or other input devices. It also ensures that the selected data source is compatible with the device and the ML model.

I may think that there are 2 data soruce here. Existing public dataset (e.g. human detection dataset stored in S3) and data (set) collected newly from own sensors. Either way, both data have to be stored in storage with label once. Once it's stored in S3, this dataset could be trained in training in the same way. We don't want to deal both dataset differently so that, any data should be stored in S3 with label once, before proceeding to Training

In the case of collecting data from sensor, we need some UI to check image data and put label on it before storing in storage.

[doyu]

Model enables users to select the ML model they want to install on the device. Users can choose from different pre-built models or upload their own custom models. Users can also select the quantization level and other parameters of the ML model. It also ensures that the selected ML model is compatible with the selected data source and device.

quantization is not yet in this phase. quantization level shoud be configured in Compiling phase.

[doyu]

Training allows users to train the selected ML model. Users can choose from different training algorithms, hyperparameters, and other settings. It also ensures that the ML model is trained properly and can achieve the desired accuracy and performance.

This is the place you'll select dataset and model for the 1st time. They sholud not be choosen in any of previous phases.

[robertmora]

Good points but IMHO each tab should provide interactive features that point to the other steps of the pipeline. For example, if I'm in the TinyMLaaS front page and a click on a device in the map, as user I would be happy to see a popup menu (i think that technically is called "context menu") that says "update device" and that brings me to the "Device" tab then for processing the specific task. Also, I should be able to remove a device from TinyMLaaS dashboard map (still through the map for example). But that's how I would see this as user of the platform. I agree that "adding" can be removed though.

Same applies for "Data" tab. From there I should be able to have a pointer that brings me to the Model part if I wish to do update the already installed model.

Each tab, still IMHO, should have a pointer/interactions to other tabs. It helps users on finding in a more easy way what they would like to do.

[doyu]

Compiling enables users to compile the selected ML model for the target device. Users can select the compilation settings, including the target architecture, optimization level, and other options. It also ensures that the compiled model is optimized for the target device and can run efficiently.

Still we shouldn't configure anything for the specific device yet here. Device specific configuration should be done in Installing phase since that's the place to generate an installable OS image. Here, Compiling means to convert a trained ML model into TFLite format. That's indepedent of CPU arch / device at all.

[robertmora]

quantization is not yet in this phase. quantization level shoud be configured in Compiling phase.

What about if I've already trained a model with two different types of quantization and these models are already stored? From where I can simply tick a box for selecting that model with re-compiling it?

[robertmora]

This is the place you'll select dataset and model for the 1st time. They sholud not be choosen in any of previous phases.

Why?

[doyu]

Installing allows users to install the compiled ML model on the target device. Users can select the device and the compiled model, and monitor the installation process. It also ensures that the compiled model is installed correctly on the target device and can run as expected.

This is the 1st place to generate an installable OS image for a specific device. This image sholud include TinyML model in it, to be used by some app specific logic running in device. There's no arch dependency in any of previous phases.

[robertmora]

Still we shouldn't configure anything for the specific device yet here. Device specific configuration should be done in Installing phase since that's the place to generate an installable OS image. Here, Compiling means to convert a trained ML model into TFLite format. That's indepedent of CPU arch / device at all.

Then perhaps we should call it "Converter" rather than "Compiling"?

[doyu]

Each tab, still IMHO, should have a pointer/interactions to other tabs. It helps users on finding in a more easy way what they would like to do.

side-bar is always visible. As explained previously, each side-bar menu item is marked done or to-be-done. This would be the indicator for which phase you are on now.

[doyu]

What about if I've already trained a model with two different types of quantization and these models are already stored? From where I can simply tick a box for selecting that model with re-compiling it?

This should be handled in registration feature in model, and probably in Compiling too

[doyu]

This is because the side-bar is the execution order from top to bottom. You shouldn't touch configuration if it's not needed yet.

[robertmora]

This should be handled in registration feature in model, and probably in Compiling too

And that's exactly what was written before.

[doyu]

The ML compiler as-a-Service concept is the core of whole archtecture.

[robertmora]

But yeah, without spamming and confusing the team, perhaps it is easier if you take the text written before and you modify at your wish and based on your view.

[doyu]

But yeah, without spamming and confusing the team, perhaps it is easier if you take the text written before and you modify at your wish and based on your view.

True! Communication is not alway easy for me ;;^^

I should have provided some desgin guide for this item. Anyway, I'm so sorry for unnecessary confusion :(

[doyu]

Summarized the above discussion in one place, https://origami-tinyml.github.io/tflm_hello_world/ui-guide.html

Feel free to update the above with a PR.

[matiasto]

Status: Borna and I began working on this.