why readSingleDataBlock and readDataList are private methods

[sccds]

Hi, I am a data engineer in a vehicle company. We use "asammdf" in python to analyze MDF file. I noticed this GO version and try to optimize the processing speed with GO. I found the signal related methods are all private. asammdf in python opened the signal reading methods. Will you consider make it public methods in the future?

[LincolnG4]

Hi @sccds

Thank you for your contribution :)

The methods readSingleDataBlock and readDataList are private because there is no need for the user/you to manage reading either a datalist block (DLBLOCK) or a datablock (DTBLOCK). The function GetChannelSample() already does it for you. When you call GetChannelFunction it will decide for you and already apply the conversion from CCBLOCK

Is there a specific reason for you to directly use readSingleDataBlock and readDataList ? I can look to put them as Public if you really need it, no worries. Just let me know.

[sccds]

hi，GetChannelSample() is Ok for the known signal name. I am exploring a method to traverse all the signals in the MDF file and store it into OLAP databases (such as Clickhouse).

[LincolnG4]

does the example below fit your needs ? it's not the most performatic or ideal, but it can help you while I finalize the implementation of a structure that stores signals in a more robust and intelligent way.

// Get channel all samples
channelMap := m.ChannelNames()
for index, channels := range channelMap {
      for _, channel := range channels {
          samples, err := m.GetChannelSample(index, channel)
          if err != nil {
              panic(err)
          }
          // put your function to save sample to store desired
              exampleFunctionSaveToOLAP(&samples)
      }
}

[sccds]

Hi I tried this,it returned panic: package not ready to read this file It looks like the error happened during reading the file:

[LincolnG4]

the package is not yet ready to read all types of data blocks. I would guess that your MDF file has a zipblock (not yet implemented). Could you send me a sample file (it can have random values/no compliance values) ? So I can debug it and I can take a look and implement the read function by Monday, It will also be useful to use this file in the unit test to help this package, if you authorize it. If that's not possible, send me an e-mail to g97santos@gmail.com and I'll investigate the file to implement the function you need and then delete the file.

[LincolnG4]

add fmt.Println(blockHeader) before the switch case and send me the output

[LincolnG4]

Take a look on development branch. I released a new structure much more optimized that runs 10x faster than current version. Now I just need the output from yours fmt.Println(blockHeader) or a sample file to release the datablock need by you.
You can take a look on the README file to see the new approach on the code.

you can use like this:

import (
    "fmt"
    "os"

    mf4 "github.com/LincolnG4/GoMDF"
)
func main() {
    file, err := os.Open("sample1.mf4")
    if err != nil {
        panic(err)
    }

    m, err := mf4.ReadFile(file)
    if err != nil {
        panic(err)
    }

file, err := os.Open("sample1.mf4")
    if err != nil {
        panic(err)
    }

    m, err := mf4.ReadFile(file)
    if err != nil {
        panic(err)
    }

// Get channel samples
    channels := m.ListAllChannels()

    for _, channel := range channels {
        samples, err := channel.Sample()
        if err != nil {
            panic(err)
        }

        fmt.Println(samples[:10])
    }
    }

[LincolnG4]

Any update ? I released a new version v1.0.0 and it has a new type of block reading SDBlocks. Also, I add a print to the block type when you receive the panic: package not ready to read this file, so you should receive the block header output that looks like "##DZ" or "## ± two letters". Than you don't need to share your file or any sensitive data. Let me know if you need help

[LincolnG4]

I suspicious that you are reading a Unsorted File, since the package is not prepared for it yet, you will receive a lot of wrong values from some channels and errors from others be aware. I'm current work on Unsorted files.

[LincolnG4]

Hi, @sccds I commited a new version in DEV to read UnsortedFiles. I will improve it yet but it can be use. Could you test it to solve your issue ? Otherwise I will close this issue due inactivity