search

SeppPenner / SparkplugNet

SparkplugNet is a library to use the Sparkplug industrial IoT (IIoT) standard in .Net. It uses MQTTnet in the background.
MIT License
83 stars 42 forks source link

Work in progress #1

Closed SeppPenner closed 3 years ago

SeppPenner commented 3 years ago

This project is not yet finished and does not compile at the moment either. I work on it and try to get it to a stable version soon.

jgmdavies commented 3 years ago

@SeppPenner Wow! Good man.

Did many others show interest? It seems more IIot than IoT so far, as you say on the 'Code' page here. Jim

SeppPenner commented 3 years ago

Did many others show interest?

Not really, I just thought that it would be a cool project :)

It seems more IIot than IoT so far, as you say on the 'Code' page here.

I have changed the description a bit.

jgmdavies commented 3 years ago

Stay cool! ;----)

I just wondered if maybe MQTTnet appeals to general IoT folk, whereas Sparkplug might be mostly on the industrial side (IIoT)? But it's interesting anyhow, and looks like it adds useful layer(s) 'on top of' MQTT.

I'm wondering if Sparkplug could be generalised to sit on top of other transport layers than MQTT, if that would interest anyone, but I haven't studied it enough yet. I suppose any alternative to MQTT would have to have similar characteristics to the ones you list on the home page here.

SeppPenner commented 3 years ago

Yeah, Sparkplug seems to be a way to define industrial IoT applications. The problem I have is that the "standard" from Eclipse is not yet well documented. I have read through the whole thing and sometimes they descriptions and examples could be better...

Theoretically it would be possible to run it on another layer except MQTT, but you need the specifications on that sub-protocol as well. I'm not an expert there, I have mainly used MQTT for that use case. I only know MQTT, AMQP and STOMP as IoT protocols and as far as I know, only MQTT supports all the needs (yet).

faldeland commented 3 years ago

Good day! I was thrilled to see a project here that would help me out :). Thank you for SparkplugNet. I was able to get it to compile and build. I was able to successfully run the unit tests after a slight change to one of them. I am new to Sparkplug. So I'm still reading through the specification. From what I know, SparkplugNet looks really good and thorough. Would you be able to comment about the level of completeness you feel it is at? I guess I would expect to see a public Publish and Subscribe method. Any thoughts about where you are, where you hoped it to be, and what is next on your list would be welcomed. I will certainly benefit from what you've done so far. Thank you!

SeppPenner commented 3 years ago

Well, I'm in the middle of progress, I would say. The core is done, but some features are still missing.

What should work:

What is not yet done:

What is ugly:

I guess, all in all I'm at around 60 percent done, just as a rough estimation. I plan to work further on this library soon (Probably on the weekend again), but it will still take some more hours (and days) to get it to a state where it can be used.

Oh, and ideas are welcome. I just did it the way I found it useful, but maybe I was missing some features or did something wrong yet and didn't recognize it or whatever.

Bazefield commented 3 years ago

Great effort. Love to see a working version that publishes metrics.

faldeland commented 3 years ago

I am 90% positive I will be using and contributing to this project for a commercial project. Before I start, do you have any plans to commit any additional work?

SeppPenner commented 3 years ago

Hi together. I was quite busy the last weeks and wasn't able to add more code here. It's planned to change some things and extend the project soon. I hope, I get some more time today to already add some more things. You will get an update once I change major things.

So, yes, I plan to work on this project over this weekend, so maybe wait until monday for contributions 😄

SeppPenner commented 3 years ago

I did some changes today:

In work:

SeppPenner commented 3 years ago

Updates today:

Todo:

I try to keep the work up again and maybe have a running initial version until the end of the week.

faldeland commented 3 years ago

FYI, I have pulled in the existing version (prior to this latest one) and was able to successfully add integration tests to connect a EoN Node connection with some Metrics to our Ignition MQTT Server and Engine. I've added some classes to support this effort. I'll bring in your latest as you push them and maintain my tests accordingly.

faldeland commented 3 years ago

Pulled in the latest, looks really good. Really coming together! To get my integration tests working, I had made some of the same modifications. I like yours better :).

SeppPenner commented 3 years ago

Nice, great to hear that you did some integration tests. I have only some tests for the message generation right now.

Changes from the last state:

Todos:

SeppPenner commented 3 years ago

I have created a NuGet package, version 0.1 as "alpha" version.

faldeland commented 3 years ago

Have you tested with an MQTT Server that supports SparkPlug? I'm having some difficulties with error messages on Start() and when I PublishMetrics(). If you have, what MQTT Server/broker are you using? Can you recommend one that might provide some level of debugging capability? I'm using Ignition.

SeppPenner commented 3 years ago

I haven't run the project against a MQTT server yet to be honest. And no, I don't know any Sparkplug compatible MQTT server to be honest. If anyone has a recommendation, I would be grateful as well. Especially for testing use cases.

SeppPenner commented 3 years ago

@faldeland Do you get the same error with Protobuf as I get?

ProtoBuf.ProtoException: "Invalid wire-type (7); this usually means you have over-written a file without truncating or setting the length; see https://stackoverflow.com/q/2152978/23354"

SeppPenner commented 3 years ago

I guess, I found the problem: The code tries to deserialize every incoming message in certain topics as ProtoBuf payload. However, the STATE messages are UTF8-encoded strings 😅

faldeland commented 3 years ago

I was able to get quite a bit farther with my tests. This is based on my limited understanding of MQTT and Sparkplug spec. I will have a PR for you to evaluate soon. I added some payload dumps to the output window so I could troubleshoot.

SeppPenner commented 3 years ago

I was able to get quite a bit farther with my tests.

Nice :)

I will have a PR for you to evaluate soon.

Ok, cool. I will check it.

For debugging MQTT messages, I use https://mqttfx.jensd.de/ and just subscribe to # (Everything) on my local broker to see what messages are sent.

I have released a new alpha version with some bug fixes, by the way.

rnitc commented 3 years ago

I haven't run the project against a MQTT server yet to be honest. And no, I don't know any Sparkplug compatible MQTT server to be honest. If anyone has a recommendation, I would be grateful as well. Especially for testing use cases.

Hi great work... looks fantastic. Hivemq Mqtt server community version comes with sparkplug as extension not sure if you can use this to debug along with ignition. https://www.hivemq.com/extension/sparkplug-influxdb-extension/

faldeland commented 3 years ago

FYI, the current unit tests are referencing the incorrect enum for Datatype (SparkplugB Namespace).

The Datatype is currently using the Protobuf enum...instead of the Sparkplug enum defined in 15.2.1. Metric Datatypes of the Sparkplug Spec.

My open pull request adds a SparkplugBDataType and SparkplugADataType enum if so desired.

NOTE: I discovered this bug when running integration tests against a Sparkplug primary application. I had converted my Metric creation to follow same pattern as the unit tests...and that's when the values stopped updating at the primary application.

jgmdavies commented 3 years ago

@SeppPenner Great work so far, many thanks. Sadly I haven't had time to participate yet, but I hope to soon. Anyway, I feel vaguely proud, after raising the subject in the 'other place' [(https://github.com/chkr1011/MQTTnet/issues/1073)] ! Best, Jim

SeppPenner commented 3 years ago

Hi great work... looks fantastic. Hivemq Mqtt server community version comes with sparkplug as extension not sure if you can use this to debug along with ignition. https://www.hivemq.com/extension/sparkplug-influxdb-extension/

I will check this soon :) @rnitc

I have already merged your pull request @faldeland. I still need to check the tests you added. From the first view, most of it seems valid. Thank you already for the contribution. However, I disagree on one or two details (But I need to check again, maybe it's correct and I just didn't get it correctly 😄).

@jgmdavies No problem, when the raw part of the package is done, there might be fine tuning or bug fixes needed in some cases, so you can contribute whenever you like :)

EDIT: I already wanted to comment yesterday, but somehow Github didn't let me. Stay tuned for updates, I will inform you here.

petedavis commented 3 years ago

Great work @SeppPenner

I have reading though the code, and I was just wondering if devices should have their own connections to the MQTT broker?

I am only just learning the protocol, but my impression was that a node contains a number of devices, and therefore the node acts as the gateway/host for the devices?

My interpretation is based on http://www.steves-internet-guide.com/introduction-to-mqtt-sparkplug-for-iiot/ image

I feel like there needs to be contract between the node and device that still allows devices to publish birth/death and metrics?

faldeland commented 3 years ago

Great work @SeppPenner

I have reading though the code, and I was just wondering if devices should have their own connections to the MQTT broker?

I am only just learning the protocol, but my impression was that a node contains a number of devices, and therefore the node acts as the gateway/host for the devices?

My interpretation is based on http://www.steves-internet-guide.com/introduction-to-mqtt-sparkplug-for-iiot/ image

I feel like there needs to be contract between the node and device that still allows devices to publish birth/death and metrics?

As I use the existing library to better understand Sparkplug and MQTT, I came to the same question.

I am using Ignition as the server/primary application and early on I was getting all sorts of errors in the logs for not conforming to the spec. This was to be expected as the library is still in progress. I am slowly working on fixing those errors as my deep dive continues. One of the errors was a message sequence numbering error. Since they were currently handled independently, the sparkplug messages ordering was unique per node and device(s). To satisfy the message sequence requirement, I modified the project slightly for each device to hold a reference to its parent node. I am making these changes in a forked project. I have not pushed those changes yet. Currently, it still creates a unique client connection for each device, but the Sparkplug messages are properly ordered. Currently no known errors in the Ignition logs.

If I read the spec correctly, each device needs to explicitly send a DDEATH when disconnected or offline. This is slightly different than the connection sequence of an EoN node.

I'll post my modifications by tomorrow EOB. The desire is to use and contribute to this project...I'm just using my fork to make keep my progress going. Perhaps my best contributions will be the integration tests :) https://github.com/faldeland/SparkplugNet

SeppPenner commented 3 years ago

I feel like there needs to be contract between the node and device that still allows devices to publish birth/death and metrics?

The problem is that I didn't see this yet in the specs (Or the specs are still unclear). I agree that it might be valid to allow devices without seperate connection to MQTT but there needs to be a common data contract and transfer technology (TCP, UDP, something else?). If someone has more information here, I would be interested.

Since they were currently handled independently, the sparkplug messages ordering was unique per node and device(s).

Mhm, that's how I read the specs to be honest :D

If I read the spec correctly, each device needs to explicitly send a DDEATH when disconnected or offline. This is slightly different than the connection sequence of an EoN node.

I'm not sure if I understand the issue here to be honest :D Can you explain this a bit more (Maybe with an example)?

faldeland commented 3 years ago

image The current implementation of SparkplugNet has a unique client connection established for node, device, application. The spec does not require a unique client connection for each device. Therefore, for each device, a PUBLISH is required for DBIRTH, DDATA, and DDEATH. In my testing with Ignition, the EoN node and the device would stay ON/Connected forever, even after I stopped/closed the connection. When I modified to explicitly PUBLISH the NDEATH and DDEATH at end of session, the primary application properly showed the data metrics as stale.

faldeland commented 3 years ago

I believe the spec assumes (or supports) that each EoN node connection is a single MQTT client connection to the MQTT Broker/Server. Current implementation of SparkplugNet has a unique MQTT client connection for each node and device(s). Testing against Ignition confirms that this is acceptable as I have not changed how nodes or devices are initialized (a unique "clientId" was required). Testing against Ignition confirmed that a Sparkplug compliant EoN node must generate a sequential "seq" # for all messages from an EoN node or any of its devices. Using Ignition for integration testing has helped immensely in clarifying the Sparkplug spec. I am beyond grateful for your contribution to SparkplugNet because it helped me get up and running so much faster than without.

SeppPenner commented 3 years ago

When I modified to explicitly PUBLISH the NDEATH and DDEATH at end of session, the primary application properly showed the data metrics as stale.

Ah okay. Now I understand the problem. What I believe is complicated, however: What if the device becomes unavailable and isn't able to publish a DDEATH message anymore? Or do I miss something here?

compliant EoN node must generate a sequential "seq" # for all messages from an EoN node or any of its devices.

Ok, I understand it now. So, currently we have different SEQ numbers for devices and nodes, but we need incrementing SEQ numbers on the node whenever a device message or a node message occurs, so only one SEQ number here.

Thank you for clearifing that to me, that was something I completely misinterpreted when reading the specs. I will test with Ignition now, too if it really helps that much :D :D

By the way, I have done some updates after your pull request @faldeland. (Mainly the logging was changed to Serilog, I chose this framework as it can be customized by the users and the log messages are only shown in debug mode and not always as before). I haven't tested if it works properly yet to be honest.

faldeland commented 3 years ago

As we test and resolve errors, it will help us better interpret the spec. This back and forth helps me to better communicate what I do and don't understand yet :).

As an EoN node, we are either the Sparkplug compliant device or hosting one or many devices that are not directly Sparkplug compliant. The node should be aware of the physical connection status of each device and should publish the DDEATH if/when it is known.

faldeland commented 3 years ago

@SeppPenner Here is an example of the logs from Ignition. The final error I'm resolving is this. I have not yet entered the correct Seq Number for the DDEATH message...and this is the error. image

SeppPenner commented 3 years ago

It took me some time, but now I'm back and want to finish a first useful version of SparkplugNet.

After I merged #2 from @faldeland, I have now adjusted the code to have the payload formats (Mainly because of different enum values in ProtoBuf and the "normal" code) seperated in the ProtoBuf data (which is only used for data transfer and afterwards directly converted to the other data format) and the externally used data.

Updates:

Todos:

What I found out:

3.4. MQTT Enabled Device(Sparkplugâ„¢) This represents any device, sensor, or hardware that directly connects to MQTT infrastructure using a compliant MQTT 3.1.1 connection with the payload and topic notation as outlined in this Sparkplugâ„¢ specification. Note that it will be represented as an EoN node in the Sparkplugâ„¢ topic payload.

This means, that you're right. Devices that have a direct MQTT connection available, are handled the same as "other" EON nodes in Sparkplug. So, devices without MQTT connection are relevant only and need to push data to the node where the node afterwards publishes the data to MQTT on behalf of the devices.

SeppPenner commented 3 years ago

Today's updates:

Todos: