search

solliancenet / MCW-Predictive-Maintenance-for-Remote-Field-Devices

Creative Commons Attribution 4.0 International
2 stars 6 forks source link

SMEs - Please add your draft review comments here #7

Open DawnmarieDesJardins opened 5 years ago

DawnmarieDesJardins commented 5 years ago

Hello, As you review this draft, please add all comments below. This will make sure everyone sees all feedback and makes conversation easier.

We appreciate your time and expertise!

olivierbloch commented 5 years ago

Main README:

olivierbloch commented 5 years ago

Workshop:

joelhulen commented 5 years ago

Thanks for the feedback, @olivierbloch. The main readme will be updated once finalized. The primary content we should focus on for the review is in the Whiteboard Design Session - Trainer guide, specifically, and the Hands-on lab. The final components of the lab will be completed tomorrow, which is primarily around creating a custom dashboard and lab cleanup.

Per your feedback:

I sent the following in an email on July 19th, regarding the ML components. Could you please read through and reply? We ended up removing the ML portion from the package due to lack of response and clear direction:

I am finalizing the architecture for the MCW and wanted to clarify some things. I understand that the primary focus is on IoT Central, and the benefits of using SaaS over PaaS. As you know, this somewhat limits any customization we can do without adding a host of additional Azure services. This leads me to the following questions and my proposed solution. I would like to see if you agree.

One of the scenario requirements is to enable alerts within IoT Central and to send out notifications using Flow. We have a trained machine learning model we created to predict impending pump failure based on telemetry. To use this trained model for near-real-time scoring, we have two different options we’ve considered:

  1. Continuously export telemetry to Event Hubs, trigger an Azure Function to send telemetry to a web service that hosts the model for scoring, then trigger Microsoft Flow to send an alert. We could not see a way to add an alert to IoT Central from the function. This option requires Event Hubs and Azure Functions, along with custom code.
  2. Create an IoT Edge device and load the trained model onto it for scoring at the edge. If a failure is predicted, send a failure prediction event that the attendees can create a trigger for in the lab. The benefit of this is that there is no need to add additional services.

Which of these options seem better to you?

Am I correct that for this MCW, we want to limit external components as much as possible so we can focus on IoT Central and the benefits it provides by being a SaaS product?

I sent a follow-up email later on the 19th in response to Vincent Hong who said he did not believe IoT Edge devices are compatible with IoT Central:

From what I understood, I thought you could connect an IoT Edge device to IoT Central and send telemetry in the same way as you would directly from a device. In fact, the IoT Central docs states the following:

From: https://docs.microsoft.com/en-us/azure/iot-central/concepts-connectivity

If your device can't use any of the supported protocols, you can use Azure IoT Edge to do protocol conversion. IoT Edge supports other intelligence-on-the-edge scenarios to offload processing to the edge from the Azure IoT Central application.

Upon further investigation, it seems like you are correct in that there’s no way to directly register an IoT Edge device. I found this repo for a bridge for connecting IoT Edge devices to IoT Central. Unfortunately, I think it requires you to set up a separate IoT Hub to register the edge device, then send the telemetry to IoT Central. The other limitation is that communication is unidirectional from the edge device to IoT Central. It cannot receive commands and forward to connected IoT devices: https://github.com/iotblackbelt/EdgeIOTCBridgeSolution

Given these limitations, it seems that using IoT Edge in the way I had hoped is dead in the water. Are there any known workarounds for getting IoT Edge devices connected to IoT Central, or should we go with option 1 below?

Due to lack of direction from the team after sending the above two emails, we have created a strictly IoT Central-focused MCW package. We do have a custom ML model we created that makes predictions based on training data we generated. The primary question is whether this MCW is the right one to add in the ML components, or if we would be better served creating another MCW that focuses more on ML along with IoT PaaS services?

olivierbloch commented 5 years ago

To the question about where to run the ML algorithm, I think we need to keep it in the Cloud as Edge is not yet supported in IoT Central and it would confuse people to have to go through hoops to make Edge work with IoT Central. For the best way to implement ML behind IoT Central, we would need to involve a couple more folks to the discussion to go into details of what’s the best way to do so as there are several options (going through Event Hubs as described below is one, going through blob storage is another). For the alert back to IoT Central, at this point you can only send a command (Direct Method) to a device using the Microsoft Flow connector. Sending a notification out (not in IoT Central) seems reasonable though. A simpler way to implement anomaly detection in the cloud is to leverage Azure Stream Analytics as described in this blog posthttps://azure.microsoft.com/en-us/blog/anomaly-detection-using-built-in-machine-learning-models-in-azure-stream-analytics/. It abstract the complexity of having to host a web app to run the model. I suggest we go this route (IoT Central -> Event Hub -> Stream Analytics with ML -> MSFT Flow

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Joel Hulen notifications@github.com Sent: Tuesday, August 6, 2019 12:43:49 PM To: solliancenet/MCW-Predictive-Maintenance-for-Remote-Field-Devices MCW-Predictive-Maintenance-for-Remote-Field-Devices@noreply.github.com Cc: Olivier Bloch (IOT) Olivier.Bloch@microsoft.com; Mention mention@noreply.github.com Subject: Re: [solliancenet/MCW-Predictive-Maintenance-for-Remote-Field-Devices] SMEs - Please add your draft review comments here (#7)

Thanks for the feedback, @olivierblochhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Folivierbloch&data=02%7C01%7COlivier.Bloch%40microsoft.com%7C73a87c012fa94f7a516608d71aa667a6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637007174313904407&sdata=8mWqfwy8zhiWiC1Xm%2FrF6LKmzJql10C8YjawWBzm%2B5E%3D&reserved=0. The main readme will be updated once finalized. The primary content we should focus on for the review is in the Whiteboard Design Session - Trainer guide, specifically, and the Hands-on lab. The final components of the lab will be completed tomorrow, which is primarily around creating a custom dashboard and lab cleanup.

Per your feedback:

I sent the following in an email on July 19th, regarding the ML components. Could you please read through and reply? We ended up removing the ML portion from the package due to lack of response and clear direction:

I am finalizing the architecture for the MCW and wanted to clarify some things. I understand that the primary focus is on IoT Central, and the benefits of using SaaS over PaaS. As you know, this somewhat limits any customization we can do without adding a host of additional Azure services. This leads me to the following questions and my proposed solution. I would like to see if you agree.

One of the scenario requirements is to enable alerts within IoT Central and to send out notifications using Flow. We have a trained machine learning model we created to predict impending pump failure based on telemetry. To use this trained model for near-real-time scoring, we have two different options we’ve considered:

  1. Continuously export telemetry to Event Hubs, trigger an Azure Function to send telemetry to a web service that hosts the model for scoring, then trigger Microsoft Flow to send an alert. We could not see a way to add an alert to IoT Central from the function. This option requires Event Hubs and Azure Functions, along with custom code.
  2. Create an IoT Edge device and load the trained model onto it for scoring at the edge. If a failure is predicted, send a failure prediction event that the attendees can create a trigger for in the lab. The benefit of this is that there is no need to add additional services.

Which of these options seem better to you?

Am I correct that for this MCW, we want to limit external components as much as possible so we can focus on IoT Central and the benefits it provides by being a SaaS product?

I sent a follow-up email later on the 19th in response to Vincent Hong who said he did not believe IoT Edge devices are compatible with IoT Central:

From what I understood, I thought you could connect an IoT Edge device to IoT Central and send telemetry in the same way as you would directly from a device. In fact, the IoT Central docs states the following:

From: https://docs.microsoft.com/en-us/azure/iot-central/concepts-connectivityhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fiot-central%2Fconcepts-connectivity&data=02%7C01%7COlivier.Bloch%40microsoft.com%7C73a87c012fa94f7a516608d71aa667a6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637007174313904407&sdata=DX04imnlyf5P%2BI5AQpGDQFDwt1yPcEGreivN53kNiGQ%3D&reserved=0

If your device can't use any of the supported protocols, you can use Azure IoT Edge to do protocol conversion. IoT Edge supports other intelligence-on-the-edge scenarios to offload processing to the edge from the Azure IoT Central application.

Upon further investigation, it seems like you are correct in that there’s no way to directly register an IoT Edge device. I found this repo for a bridge for connecting IoT Edge devices to IoT Central. Unfortunately, I think it requires you to set up a separate IoT Hub to register the edge device, then send the telemetry to IoT Central. The other limitation is that communication is unidirectional from the edge device to IoT Central. It cannot receive commands and forward to connected IoT devices: https://github.com/iotblackbelt/EdgeIOTCBridgeSolutionhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fiotblackbelt%2FEdgeIOTCBridgeSolution&data=02%7C01%7COlivier.Bloch%40microsoft.com%7C73a87c012fa94f7a516608d71aa667a6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637007174313914408&sdata=vd736quV1Wg%2F789sas%2F9p%2FWezhqMxs4jxUw5cQ%2FosnY%3D&reserved=0

Given these limitations, it seems that using IoT Edge in the way I had hoped is dead in the water. Are there any known workarounds for getting IoT Edge devices connected to IoT Central, or should we go with option 1 below?

Due to lack of direction from the team after sending the above two emails, we have created a strictly IoT Central-focused MCW package. We do have a custom ML model we created that makes predictions based on training data we generated. The primary question is whether this MCW is the right one to add in the ML components, or if we would be better served creating another MCW that focuses more on ML along with IoT PaaS services?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsolliancenet%2FMCW-Predictive-Maintenance-for-Remote-Field-Devices%2Fissues%2F7%3Femail_source%3Dnotifications%26email_token%3DAAJWOQHOAT76PDCHFQ5SRKLQDHH7LA5CNFSM4IHH2XAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WH5AY%23issuecomment-518815363&data=02%7C01%7COlivier.Bloch%40microsoft.com%7C73a87c012fa94f7a516608d71aa667a6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637007174313914408&sdata=zi61M9DRXDnBQT1QGGp4pgCwi1yPSN3KOnOPrsP0P%2BY%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAJWOQAEQFCUHCIIWXA4T3TQDHH7LANCNFSM4IHH2XAA&data=02%7C01%7COlivier.Bloch%40microsoft.com%7C73a87c012fa94f7a516608d71aa667a6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637007174313924417&sdata=0WbWgYaK9LiyIOwU7hkzJKz%2FEgZIL9mUwiLc448AJxs%3D&reserved=0.

joelhulen commented 5 years ago

Given your suggestion of using IoT Central -> Event Hub -> Stream Analytics with ML -> MSFT Flow, how would you like to proceed? Should we wait to involve others for feedback? This approach will require a change to how we're tracking and alerting on anomalies in the hands-on-lab, where the process is done entirely in IoT Central. Should we keep the lab as-is and add your suggested flow to the WDS? The WDS can be expanded to discuss this method of anomaly detection and a more advanced option that uses a custom trained model. I did not add these ML components in the WDS because I wasn't sure how they fit in with the expressed desire to focus on SaaS and IoT Central, specifically.

olivierbloch commented 5 years ago

As of now the process doesn’t detect anomalies, it triggers an alert when a threshold is set manually. There is no Machine Learning Anomaly detection involved. The very promise of the lab is to do predictive maintenance, which by definition involves machine learning Do you want to jump on a call tomorrow to discuss further and close on this?

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Joel Hulen notifications@github.com Sent: Wednesday, August 7, 2019 1:46:25 PM To: solliancenet/MCW-Predictive-Maintenance-for-Remote-Field-Devices MCW-Predictive-Maintenance-for-Remote-Field-Devices@noreply.github.com Cc: Olivier Bloch (IOT) Olivier.Bloch@microsoft.com; Mention mention@noreply.github.com Subject: Re: [solliancenet/MCW-Predictive-Maintenance-for-Remote-Field-Devices] SMEs - Please add your draft review comments here (#7)

Given your suggestion of using IoT Central -> Event Hub -> Stream Analytics with ML -> MSFT Flow, how would you like to proceed? Should we wait to involve others for feedback? This approach will require a change to how we're tracking and alerting on anomalies in the hands-on-lab, where the process is done entirely in IoT Central. Should we keep the lab as-is and add your suggested flow to the WDS? The WDS can be expanded to discuss this method of anomaly detection and a more advanced option that uses a custom trained model. I did not add these ML components in the WDS because I wasn't sure how they fit in with the expressed desire to focus on SaaS and IoT Central, specifically.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsolliancenet%2FMCW-Predictive-Maintenance-for-Remote-Field-Devices%2Fissues%2F7%3Femail_source%3Dnotifications%26email_token%3DAAJWOQAQHC3J74ZVG2TXSTLQDMYCDA5CNFSM4IHH2XAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ZVB4Y%23issuecomment-519262451&data=02%7C01%7COlivier.Bloch%40microsoft.com%7Cd1f343c185e14f3f837408d71b785082%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637008075867467516&sdata=mIvYo2Q2VL8I6HAh%2FRDum9UDEU6HM3fKKeH%2B92sq%2BtU%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAJWOQEUQ73CGVMZ5O3MQ6TQDMYCDANCNFSM4IHH2XAA&data=02%7C01%7COlivier.Bloch%40microsoft.com%7Cd1f343c185e14f3f837408d71b785082%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637008075867477513&sdata=nDioIDWL7PIH41VEDM7QJZJE3hEwwxPkgOyIjOpyjAg%3D&reserved=0.

joelhulen commented 5 years ago

Agreed with your assessment. We even have a custom-trained model we built for this lab. However, the approach we wanted to take with it bypassed a lot of IoT Central's capabilities, which is why we posed the questions we did. I would like to discuss this in a call so we can nail down the ideal approach, with your goals in mind.