How to contribute

[kegesch]

I just found your project and wondered if I can help you somehow since I am planning to use your library.

[fkohlgrueber]

Hi Jonas,

thanks for your interest in the project. It's great to hear that you're planning to use the library! I'm always looking for contributors and would appreciate any help.

I'm currently developing this library in my free time, so progress is quite slow. I'm also planning to do a series of blog posts about it since there's a lot of interesting things to talk about.

If you want to help, there's a couple of things I could imagine off the top of my head:

• Update / write wikipedia articles for sml. Currently, there's only an article on German wikipedia and many links within it are outdated. It's a bit weird to link to German wikipedia in a project that uses English. Having an entry in the English wikipedia would be great, even if it's a really basic one.
• Review code. I have already implemented more than what's published and once I've done some cleanup, I'd appreciate feedback.
• Tell me about your use-case. That's possibly the most interesting part for me right now. My goal is to use the library on an esp32c3 microcontroller and using that one to build a DIY "smart meter". I'd like to know what you're planning to do, so feel free to tell me. Depending on your use-case, I can adapt the focus of the project etc.

I'm also open to code contributions, but I think it doesn't make much sense before I've published my current state of work. I'll try to do that soon.

We can also talk about the project in person (discord call etc.). Just let me know if you're interested.

[kegesch]

Hi Felix, glad to know that you are still working on it. I am planning to use the library in a project that i am working on in my free time as well, so don't worry about progress. About my use-case. Some time ago (prior studying cs) I created a small "web app" that collects the current total of a counter every night, saves in a DB and visualizes it. Now there is a new "feature-request" and I wanted to rewrite the part of reading the data from the counter (basically this solarbeam) since i made this in retrospective quite poorly and insecure. This app runs on a raspberry pi which is connected via an IR-Usb dongle to an EHZ.

If i find the time i could write an english article but i think this will not be my main goal. But you can definetely add me as a reviewer for code reviews. When you have the main structure of the project, i could definetely help you with coding. Just let me know, then we can for sure have a short call.

[felixwrt]

Hi Jonas,

it's been a while, but I've been working on sml-rs and just published a new release. I've implemented parsing of sml messages, which should make the library usable for your project. Feel free to check it out and I'm looking forward to feedback!

[kegesch]

Hi Felix, I finally had the time and the setup to test it.

I am reading from my smartmeter with the serialport library and collecting the messages based on the start and end sequences. However, the parser always yields an TlfInvalidTy error. Do you have any idea why?

This is the data:

data: [1B, 1B, 1B, 1B, 01, 01, 01, 01, 76, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 07, 00, 0C, 06, 5D, 75, F0, 62, 00, 62, 00, 72, 63, 01, 01, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 76, 01, 01, 07, 00, 0C, 0D, 0A, 27, 50, 0B, 09, 01, 45, 4D, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 48, 00, 00, 54, 59, BF, 01, 01, 63, 59, CF, 00, 76, 07, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 0C, 06, 5D, 75, F1, 62, 00, 62, 00, 72, 63, 07, 01, 77, 01, 0B, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 09, 01, 45, 4D, 48, 00, 00, 54, 59, BF, 07, 01, 00, 62, 0A, FF, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, FF, 72, 62, 01, 65, 0D, 0A, 67, 52, 77, 77, 07, 81, 81, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, C7, 82, 03, FF, 01, 01, 01, 01, 04, 45, 4D, 48, 01, 77, 07, 01, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 09, FF, 01, 01, 01, 01, 0B, 09, 01, 45, 4D, 48, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 54, 59, BF, 01, 77, 07, 01, 00, 01, 08, 00, FF, 64, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 01, 80, 01, 62, 1E, 52, FF, 56, 00, 0E, 25, 62, C5, 01, 77, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 07, 01, 00, 01, 08, 01, FF, 01, 01, 62, 1E, 52, FF, 56, 00, 0E, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 25, 62, C5, 01, 77, 07, 01, 00, 01, 08, 02, FF, 01, 01, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 62, 1E, 52, FF, 56, 00, 00, 00, 00, 00, 01, 77, 07, 01, 00, 10, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 07, 00, FF, 01, 01, 62, 1B, 52, FF, 55, 00, 00, 01, D7, 01, 77, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 07, 81, 81, C7, 82, 05, FF, 01, 72, 62, 01, 65, 0D, 0A, 67, 52, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 01, 01, 83, 02, 13, 39, 9D, B7, 17, F6, 3B, 22, 8D, 94, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, F4, BB, 36, FC, A6, 90, 46, FB, DD, 52, 5D, CC, 2C, F1, 9A, EF, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 6C, A2, 2A, 6E, C8, 44, 06, 36, 52, D9, D6, 3B, A7, 83, 45, E1, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 5F, FC, 11, 14, 87, D1, 01, 01, 01, 63, 6B, A3, 00, 76, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 07, 00, 0C, 06, 5D, 75, F4, 62, 00, 62, 00, 72, 63, 02, 01, 71, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 01, 63, 8F, 15, 00, 00, 00, 00, 1B, 1B, 1B, 1B, 1A, 03, 9F, 00]

And this is the parser result:

Err(
    InvalidTlf(
        TlfInvalidTy,
    ),
)

[felixwrt]

Hi, that's interesting. I (on purpose) only implemented the parts of the SML spec that are actually used in real world meters, based on the data from the libsml-testing repo. Seems like your meter uses something special.

I can take a quick look at the data you provided. I'll get back to you within the next few days.

[felixwrt]

I can't reproduce the TlfInvalidTy error you were seeing. Using the data you've provided, I'm getting a CRC mismatch while decoding the transport:

    Err(
        InvalidMessage {
            checksum_mismatch: (
                159,
                15448,
            ),
            end_esc_misaligned: false,
            num_padding_bytes: 3,
        },
    ),

Looking at the data, it seems like there's a lot of 00 bytes in there, more than usual for sml data. Did you modify the data? If not, it looks like your data collection doesn't work reliably. Whats the smart meter model you're using?

[felixwrt]

Oh, another thing: If you're getting TlfInvalidTy using the data posted above, I'm assuming you used sml_rs::parser::complete::parse(&data);, right? That API expects the transport layer to be unpacked already (so that the data doesn't start with 1B1B1B1B01010101 anymore). You can unpack the transport using sml_rs::transport::Decoder. See the examples folder in the repo for a complete example of how to unpack the transport layer and parse the data.

I've wanted to add an example showing how to use sml-rs with serialport-rs to read sml data from a serial port. If you're at it and still want to contribute, creating such an example would be great. Otherwise, if you could share your code, I could use that to create the example myself.

Looking forward to hear from you!

[kegesch]

You are absolutely right. I read the data into a too large buffer which caused the extra zero bytes inbetween. I also only used the parse-method, but using the decode function prior parsing works as intended.

Is there a reason why you split the process into decoding and aprsing in the public API? Or in other words: I would have assumend that the parse function is all you need as a user and the decoding would happen internally.

I can share my current code if you want, then you can add it to your codebase in the way you prefer. No guarantees that this is the best solution :D

Example with serialport

main.rs

use sml_rs::parser::complete::{parse};
use sml_rs::transport::{decode};

use serialport::{StopBits, Parity};

fn main() {
    let ports = serialport::available_ports().expect("No ports found!");

    println!("Available Ports");
    for p in ports {
        println!("{}", p.port_name);
    }

    const DEVICE: &str = "/dev/ttyUSB1";

    println!("Connecting to {}", DEVICE);
    let port = serialport::new(DEVICE, 9_600)
        .stop_bits(StopBits::One)
        .parity(Parity::None)
        .timeout(Duration::from_millis(5000))
        .open().expect("Failed to open port");

    let mut port = BufReader::new(port);
    let mut data: Vec<u8> = vec![];
    let mut has_message_started = false;
    let mut message_end_index: Option<usize> = None;

    const START_SEQ: [u8; 8] = [0x1b, 0x1b, 0x1b, 0x1b, 0x01, 0x01, 0x01, 0x01];
    const END_SEQ: [u8; 5] = [0x1b, 0x1b, 0x1b, 0x1b, 0x1a];

    // Each loop iteration reads one byte, adds it to the aggregated data collection. Then the data which was collected until the iteration is analysed if it contains the start and the end-sequences.
    loop {   
        let mut serial_buf: Vec<u8> = vec![0; 1];

        // read one byte and add to data vector
        port.read(serial_buf.as_mut_slice()).expect("Found no data!");
        data.append(&mut serial_buf);

        // if the SML message has not yet started look for the start sequence in the message and cut the current collected data at the beginning of the start sequence
        if !has_message_started {
            let index_of_start = find_subsequence(data.as_slice(), &START_SEQ);
            if let Some(index) = index_of_start {
                has_message_started = true;
                data = (&data[index..]).to_vec();
            }
        }

        // look for the the end sequence in the data, if it was not already found
        if message_end_index.is_none() {
            message_end_index = find_subsequence(data.as_slice(), &END_SEQ);
        }

        if let Some(index) = message_end_index {
            if data.len() == index + 8 {
                if has_message_started {
                    // if the end-sequence with the checksum is aggregated in the data and the data contains a start sequence the data can be decoded and parsed
                    handle_data(data.as_slice());

                    // reset state
                    data.clear();
                    has_message_started = false;
                    message_end_index = None;
                }
            }
        }
    }  
}

fn handle_data(data: &[u8]) {
    let decoded = decode(data);
    for d in decoded {
        if let Ok(decoded_data) = d {
            let slice = decoded_data.as_slice();
            let parse_result = parse(slice);
            println!("{:#?}", parse_result)
        }
    }
}

fn find_subsequence(haystack: &[u8], needle: &[u8]) -> Option<usize> {
    haystack.windows(needle.len()).position(|window| window == needle)
}

dependencies in cargo.toml

[dependencies]
serialport = "4.2.0"
sml-rs = { version = "0.2.1", features = ["alloc"]}

[felixwrt]

Is there a reason why you split the process into decoding and aprsing in the public API? Or in other words: I would have assumend that the parse function is all you need as a user and the decoding would happen internally.

Yeah, I can see that this is confusing. I'm not sure on the "higher-level" APIs yet. I could imagine several "higher-level" aspects in an API:

• Combining transport decoding and parsing
• Expecting an sml transmission to consist of exactly three messages (OpenResponse, GetListResponse, CloseResponse - in that order), which is the scheme of all real-world data I've seen so far.
• Data structures reduced to commonly-used items (e.g. without the signature key that no meter uses)
• High-Level ListEntry type that for example provides values as floats with the correct scalar applied
• ...

Long story short: I'm still unsure how to design more convenient ("high-level") APIs and that's why currently, there's only separate "building block" APIs for transport decoding and parsing.

I can share my current code if you want, then you can add it to your codebase in the way you prefer. No guarantees that this is the best solution :D

Your code does more than it needs to do. Detecting the start / end of a transmission is handled by the transport layer implementation in sml-rs. I've created an example for how to read sml data from a serial port. See this PR and let me know if you have questions.

sml-rs = { version = "0.2.1", features = ["alloc"]}

Btw: The std and alloc features are active by default in sml-rs. You can simply use sml-rs = "0.2.1".