Ziptype num:59 for header ##DZ

[sccds]

Hi, I transfered to some others projects in the past month. Recently I get some free time and try to optimize this MDF file project. I download the dev branch source code and try to analyze some source code.

I am using this code to iterate groups and iterate each signals values.

    for gid, gp := range mdfFile.DataGroups {
        for _, cg := range gp.ChannelGroup {
            //fmt.Printf("gid[%d]: gp[%v], id[%d], cg[%v], Data[%v] \n", gid, gp, id, cg)
            var i int = 0
            fmt.Printf("group: %d, rowSize: %d cycleCount: %d \n", gid, cg.Block.RowSize(), cg.Block.Data.CycleCount) //**
            if cg.Block.Data.CycleCount <= 0 {                                                                        
                continue
            }
            var chid int = 0
            for cn, ch := range cg.Channels {
                fmt.Println()
                fmt.Println("\t", "new channel:", chid, cn)
                chid++
                sam, er := ch.Sample()
                if er != nil {
                    fmt.Println("\t", er)
                    continue
                }
                fmt.Println("\t\t", len(sam), sam)
            }

            i++
        }

    }

Issues will happen when header is "##DZ" and zipType is 59, which is not supported in your code.

[LincolnG4]

Hi @sccds 😊 First, to iterate over all channels you can use the sample code in README.

m, err := mf4.ReadFile(file, &mf4.ReadOptions{MemoryOptimized: false})
if err != nil {
   panic(err)
}

// Get channel samples
channels := m.ListAllChannels()
for _, channel := range channels {
   samples, err := channel.Sample()
   if err != nil {
    panic(err)
   }

   fmt.Println(channel.DataGroupIndex, channel.Name, samples)
}

Second, the zipType==59 should not exist. The ASAM protocol covers only 2 types (Deflate and Transposition + Deflate) as you can see here :

So there are a few hypotheses:

• There is a bug in the code that when reading the binary of the mdf file instead of turning it into 0 or 1, it reads the wrong space and turns it into 59.
• Your MDF file writer is using 59 and is wrong (in terms of the ASAM protocol). It could be a bug or 59 is internal code for a specific decompression, which would require a customized decompression.

I'll take a look at the code and see if I can find any problems. I've already tested with some files with ##DZ and didn't find this problem.

On the other hand, you can debug the file using a tool like Canalyzer's MDF Validator or Python's ASAMMDF ? MDF Validator is a free and easy-to-use tool.

1. Download MDF Validator and open the file
2. Click on the zipped block of data in which you found type 59
3. Check the value of the data type

so we can find out if it's a bug in my code, a writing bug in your file generator or if you need a custom decompression that we can add to the code.

[sccds]

Use the sample code meet panics. So I try to iterate group and channls comparing with Python asammdf result. Python asammdf is OK to read the signal values.

In my MDF file, not all groups meet errors. The first several groups from the beginning have error. When meet header "##DZ" not all the result recognized as wrong zip type. When reading groups and channel, the error looks like these log:

header is:  ##HL
header is:  ##DL
header is:  ##DZ
ziptype: 0
header is:  ##DT
dt, pos: 4317 4365 47250
... some value
header is:  ##DZ
ziptype: 0
header is:  ##DT
dt, pos: 4317 4365 47250
... some value
header is:  ##DZ
ziptype: 0
header is:  ##DT
dt, pos: 4317 4365 47250
... some value
header is:  ##DZ
ziptype: 0
header is:  ##DT
dt, pos: 4317 4365 47250
... some value
header is:  ##DZ
ziptype: 0
header is:  ##DT
dt, pos: 4317 4365 47250
... some value
header is:  ##DZ
ziptype: 59
         invalid decompress type 59

I will try to use MDF validator and give you the feedback later.

[LincolnG4]

Thanks for your feedback. I will take a look into the code to see if I can found the issue.

"I will try to use MDF validator and give you the feedback later." It would be great! Thanks

[LincolnG4]

Hi @sccds

Please try the development branch Use channel.PrintProperties() to debug

    for index, channel := range channels {

        fmt.Println(index, channel.Name)
        samples, err := channel.Sample()
        if err != nil {
                        channel.PrintProperties()
            panic(err)
        }
        .....
    }

I changed the way the parameters are loaded from DZBlocks to see if that fixes your problem.
I fixed some bugs and found a weird comportament in the transpose decompress functions that I couldn't fix yet. Also note that ASAMMDF ignores if the zypType is greater than 1. So I remove the error and treat in same way of ASAMMDF to see if works in your code

let me know the results

[sccds]

Hi I use your code, and get error.

0 time
&{Header:{ID:[35 35 67 78] Reserved:[0 0 0 0] Length:160 LinkCount:8} Link:{Next:4496 Composition:0 TxName:4152 SiSource:4344 CcConvertion:4400 Data:0 MdUnit:4184 MdComment:4216 AtReference:0 DefaultX:[0 0 0]} Data:{Type:2 SyncType:1 DataType:2 BitOffset:0 ByteOffset:0 BitCount:64 Flags:2048 InvalBitPos:0 Precision:0 Reserved:0 AttachmentCount:0 ValRangeMin:0 ValRangeMax:0 LimitMin:0 LimitMax:0 LimitExtMin:0 LimitExtMax:0}}panic: invalid block ID

I am still contact IT to help install the mdf validator, it may take several days.

[LincolnG4]

Hi, I've added a few more debug messages to the development branch, could you run them and send me the errors? Please send all the error messages so I know where exactly the code is failing.

[sccds]

0 time
&{Header:{ID:[35 35 67 78] Reserved:[0 0 0 0] Length:160 LinkCount:8} Link:{Next:4496 Composition:0 TxName:4152 SiSource:4344 CcConvertion:4400 Data:0 MdUnit:4184 MdComment:4216 AtReference:0 DefaultX:[0 0 0]} Data:{Type:2 SyncType:1 DataType:2 BitOffset:0 ByteOffset:0 BitCount:64 Flags:2048 InvalBitPos:0 Precision:0 Reserved:0 AttachmentCount:0 ValRangeMin:0 ValRangeMax:0 LimitMin:0 LimitMax:0 LimitExtMin:0 LimitExtMax:0}}panic: invalid block ID: expected ##DZ, got: ##DT. Read block: {ID:[35 35 68 84] Reserved:[0 0 0 0] Length:4749 LinkCount:0}

goroutine 1 [running]: