search

Azure / azure-iot-operations

The official repo for Azure IoT Operations.
MIT License
20 stars 16 forks source link

[feature] Add support for windows network drives as data pipeline source #37

Open derbl4ck opened 4 months ago

derbl4ck commented 4 months ago

Is your feature request related to a problem? Please describe. In our manufacturing enterprise we have machines that generate .XML Files to local disk containing detailed measurings which are interesting for further analysis in Microsoft Fabric or Azure. Those detailed measurings are right now not exposed via OPC UA or other protocolls which is the reason, why this "Fileshare"-Broker is so interesting.

Describe the solution you'd like We should add a new Data Source in Azure IoT Operations Data Pipelines called "Fileshare" supporting smb network drives from the windows manufacturing machines. As trigger we could have two options: Time interval as well as file creation.

Describe alternatives you've considered Using a UI-managed paid 3rd-party tool to extract data out of the local files and send via MQTT to a message broker.

Additional context To build the needed container we can rely on common samba tools like https://pkgs.alpinelinux.org/package/edge/main/x86/samba-client taking hostname, username and password as argument to establish a connection to the fileshare. To bring those files to MQTT we could use https://github.com/sidleal/mqttout with https://github.com/elastic/beats/tree/main/filebeat.

See also https://feedback.azure.com/d365community/idea/76500483-ffc1-ee11-92bc-6045bd7aea25

ruonke commented 4 months ago

"fileshare"-broker sounds super complicated for what you are trying to do.

I would just use something like this to push the XML data to MQTT

derbl4ck commented 4 months ago

I would just use something like this to push the XML data to MQTT

Hi @ruonke, thank you for the golang xml package. For someone who is able to write go and has direct access to those systems, this for sure would be one possible and less-overhead solution. Since the solution must be usable for our manufacturing engineers, which do not have much IT expertise or system access, it should be configurable within the Azure IoT Operations WebUI.

Right now those manufacturing engineers are using additional paid 3rd-party tools like Telit deviceWISE to achieve the same.

meenag16 commented 4 months ago

The recommended approach here would be the following approach: With a remote shared folder, use the NFS provisioner and directly connect to the NFS storage from a linux container (AKS-EE cluster with NFS storage CSI), have a pod with NFS PVC, and use XML to MQTT logic as mentioned in this thread.

https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-edge-howto-use-storage-nfs

derbl4ck commented 4 months ago

(AKS-EE cluster with NFS storage CSI), have a pod with NFS PVC, and use XML to MQTT logic as mentioned in this thread.

Great Idea @meenag16! This would result in the following picture: xmltomqtt

Assuming the OT Engineer in the picture cant code GoLang and does not know much about containers, this brings up the following question:

How to add "custom connectors" to allow the OT Engineer to work with the Azure IoT Operations UI only? (Number2 in the visual) If this is possible, I could/would provide this XML / File to MQTT connector (as open source) 🙂

(Optional) It would be amazing to also bring up the possibility to manage the PVCs via IoT Ops UI. Maybe we can find a way to enable custom plugins. (This would remove the need for number 1 in the visual)

meenag16 commented 4 months ago

We are looking into this, and I will get back to you soon!