Error when creating more than 1 channel group or more than 1 data group.

[chrisakira]

I'm having a problem when trying to write more than one signal to an MF4 file.

The first data group always gives me an error on the cn_byte_offset, and the second one is created normally, but the data is always wrong.

If I use this code to create only one data group and one channel group, it writes perfectly. It only gives me an error when trying to write more than one.

This is my current code, and I wrote it in C++ using CMake to build the project.

`

include

include

include

include

include

include "isourceinformation.h"

include "iattachment.h"

include "ichannelgroup.h"

include "idatagroup.h"

include "ievent.h"

include "ifilehistory.h"

include "mdffactory.h"

include "mdfreader.h"

include "mdfwriter.h"

include "mdflogstream.h"

include

include

using namespace std::this_thread; using namespace std::chrono_literals; using namespace std::filesystem; using namespace mdf; using namespace std;

int main( ) { auto currentTime = std::chrono::system_clock::now(); auto duration = currentTime.time_since_epoch(); std::chrono::nanoseconds unixTime = std::chrono::duration_cast(duration); // Busca o Unix atual em Nano segundos uint64_t time_stamp = unixTime.count();

path mdf_file("../mf4_files/");
mdf_file.append("unsigned.mf4");

auto writer = MdfFactory::CreateMdfWriter(MdfWriterType::Mdf4Basic); 
mdf->Header();

writer->Init(mdf_file.string());
auto *header = writer->Header();
auto *history = header->CreateFileHistory();
history->Description("Test data types");
history->ToolName("MdfWrite");
history->ToolVendor("ACME Road Runner Company");
history->ToolVersion("1.0");
history->UserName("Ingemar Hedvall"); 

auto *data_group = header->CreateDataGroup();
auto *data_group2 = header->CreateDataGroup();
cout << meas_list.size() << endl;
auto *group1 = data_group->CreateChannelGroup();
group1->Name("Unsigned");
auto *group2 = data_group2->CreateChannelGroup();
group2->Name("Potato");

auto *ch1 = group1->CreateChannel();
ch1->Name("Intel8");
ch1->Type(ChannelType::FixedLength);
ch1->Sync(ChannelSyncType::None);
ch1->DataType(ChannelDataType::UnsignedIntegerLe);
ch1->DataBytes(1);

auto *ch2 = group1->CreateChannel();
ch2->Name("Intel16");
ch2->Type(ChannelType::FixedLength);
ch2->Sync(ChannelSyncType::None);
ch2->DataType(ChannelDataType::UnsignedIntegerLe);
ch2->DataBytes(2);

auto *ch3 = group1->CreateChannel();
ch3->Name("Intel32");
ch3->Type(ChannelType::FixedLength);
ch3->Sync(ChannelSyncType::None);
ch3->DataType(ChannelDataType::UnsignedIntegerLe);
ch3->DataBytes(4);

auto *ch4 = group1->CreateChannel();
ch4->Name("Intel64");
ch4->Type(ChannelType::FixedLength);
ch4->Sync(ChannelSyncType::None);
ch4->DataType(ChannelDataType::UnsignedIntegerLe);
ch4->DataBytes(8);

auto *ch5 = group2->CreateChannel();
ch5->Name("Motorola8");
ch5->Type(ChannelType::FixedLength);
ch5->Sync(ChannelSyncType::None);
ch5->DataType(ChannelDataType::UnsignedIntegerBe);
ch5->DataBytes(1);

auto *ch6 = group2->CreateChannel();
ch6->Name("Motorola16");
ch6->Type(ChannelType::FixedLength);
ch6->Sync(ChannelSyncType::None);
ch6->DataType(ChannelDataType::UnsignedIntegerBe);
ch6->DataBytes(2);

auto *ch7 = group2->CreateChannel();
ch7->Name("Motorola32");
ch7->Type(ChannelType::FixedLength);
ch7->Sync(ChannelSyncType::None);
ch7->DataType(ChannelDataType::UnsignedIntegerBe);
ch7->DataBytes(4);

auto *ch8 = group2->CreateChannel();
ch8->Name("Motorola64");
ch8->Type(ChannelType::FixedLength);
ch8->Sync(ChannelSyncType::None);
ch8->DataType(ChannelDataType::UnsignedIntegerBe);
ch8->DataBytes(8);

writer->PreTrigTime(0);
writer->InitMeasurement();

auto tick_time = time_stamp;
writer->StartMeasurement(tick_time);

uint64_t value;
for (size_t sample = 0; sample < 100; ++sample)
{
    value = static_cast<uint64_t>(sample);
    ch1->SetChannelValue(value);
    ch2->SetChannelValue(value);
    ch3->SetChannelValue(value);
    ch4->SetChannelValue(value); 
    writer->SaveSample(*group1, tick_time); 
    tick_time += 1'000'000'000;
}

writer->StopMeasurement(tick_time);
writer->FinalizeMeasurement();

writer->StartMeasurement(tick_time);
value = 0;
for (size_t sample = 0; sample < 100; ++sample)
{
    value = static_cast<uint64_t>(sample); 
    ch5->SetChannelValue(value);
    ch6->SetChannelValue(value);
    ch7->SetChannelValue(value);
    ch8->SetChannelValue(value); 
    writer->SaveSample(*group2, tick_time);
    tick_time += 1'000'000'000;
}
writer->StopMeasurement(tick_time);
writer->FinalizeMeasurement();

} `

[ihedvall]

Well this might be little difficult to explain but you can only work with one data group (DG) for each measurement. The writer always add the sample to the last data group. The reason is that the samples are appended to the end of the file.

You should be able to append samples from several channel groups as long as you add the samples in chronological (time) order.

In your code, you should move your code data_group2 and group2 code, to after the first FinilizeMeasurement. Then init the measurement and do a new measurement.

As an alternative is to skip the data_group2, create the group2 from the data_group(1) and move the second for loop of samples to after the first for loop of samples.

You can also mix samples from several channel groups but you can only handle one data group at the time. Data Group(Measurement) consist of one or more Channnel Groups (Sub-mesurements).

I can do an unit test tomorrow because it is to late today.

Note that InitMeasurement and FinalizeMeasurement have a one to one relation. InitMeasurement opens the file while FinalizeMeasurement closes the file.

[ihedvall]

I have tested the below unit test and it works as expected. Its a file with 2 measurements with 2 sub-measurements. Note that a generic MDF Reader is relative easy to do but an MDF Writer is harder to do generic. The MDF4 Basic Writer is a typical data logger that append samples as the arrive in time.

There have been discussion to do other writer type as CAN Loggers or MDF Batch Converters. So if you have any use case/requirements, please reply.

TEST_F(TestWrite, Mdf4Multi) {
  if (kSkipTest) {
    GTEST_SKIP();
  }
  path mdf_file(kTestDir);
  mdf_file.append("multi.mf4");

  auto writer = MdfFactory::CreateMdfWriter(MdfWriterType::Mdf4Basic);
  writer->Init(mdf_file.string());
  auto* header = writer->Header();
  auto* history = header->CreateFileHistory();
  history->Description("Test data types");
  history->ToolName("MdfWrite");
  history->ToolVendor("ACME Road Runner Company");
  history->ToolVersion("1.0");
  history->UserName("Ingemar Hedvall");
  { // Measurement 1
    auto* data_group = header->CreateDataGroup();
    auto* group1 = data_group->CreateChannelGroup();
    group1->Name("Intel");

    auto* ch1 = group1->CreateChannel();
    ch1->Name("Intel32");
    ch1->Type(ChannelType::FixedLength);
    ch1->Sync(ChannelSyncType::None);
    ch1->DataType(ChannelDataType::FloatLe);
    ch1->DataBytes(4);

    auto* ch2 = group1->CreateChannel();
    ch2->Name("Intel64");
    ch2->Type(ChannelType::FixedLength);
    ch2->Sync(ChannelSyncType::None);
    ch2->DataType(ChannelDataType::FloatLe);
    ch2->DataBytes(8);

    auto* group2 = data_group->CreateChannelGroup();
    group2->Name("Motorola");

    auto* ch3 = group2->CreateChannel();
    ch3->Name("Motorola32");
    ch3->Type(ChannelType::FixedLength);
    ch3->Sync(ChannelSyncType::None);
    ch3->DataType(ChannelDataType::FloatBe);
    ch3->DataBytes(4);

    auto* ch4 = group2->CreateChannel();
    ch4->Name("Motorola64");
    ch4->Type(ChannelType::FixedLength);
    ch4->Sync(ChannelSyncType::None);
    ch4->DataType(ChannelDataType::FloatBe);
    ch4->DataBytes(8);

    writer->PreTrigTime(0);
    writer->InitMeasurement();

    auto tick_time = TimeStampToNs();
    writer->StartMeasurement(tick_time);

    for (size_t sample = 0; sample < 100; ++sample) {
      auto value = static_cast<double>(sample) + 0.23;
      ch1->SetChannelValue(value);
      ch2->SetChannelValue(value);
      ch3->SetChannelValue(value);
      ch4->SetChannelValue(value);

      writer->SaveSample(*group1, tick_time);
      writer->SaveSample(*group2, tick_time);
      tick_time += 1'000'000'000;
    }
    writer->StopMeasurement(tick_time);
    writer->FinalizeMeasurement();
  }
  { // Measurement 2
    auto* data_group = header->CreateDataGroup();
    auto* group1 = data_group->CreateChannelGroup();
    group1->Name("Intel");

    auto* ch1 = group1->CreateChannel();
    ch1->Name("Intel32");
    ch1->Type(ChannelType::FixedLength);
    ch1->Sync(ChannelSyncType::None);
    ch1->DataType(ChannelDataType::FloatLe);
    ch1->DataBytes(4);

    auto* ch2 = group1->CreateChannel();
    ch2->Name("Intel64");
    ch2->Type(ChannelType::FixedLength);
    ch2->Sync(ChannelSyncType::None);
    ch2->DataType(ChannelDataType::FloatLe);
    ch2->DataBytes(8);

    auto* group2 = data_group->CreateChannelGroup();
    group2->Name("Motorola");

    auto* ch3 = group2->CreateChannel();
    ch3->Name("Motorola32");
    ch3->Type(ChannelType::FixedLength);
    ch3->Sync(ChannelSyncType::None);
    ch3->DataType(ChannelDataType::FloatBe);
    ch3->DataBytes(4);

    auto* ch4 = group2->CreateChannel();
    ch4->Name("Motorola64");
    ch4->Type(ChannelType::FixedLength);
    ch4->Sync(ChannelSyncType::None);
    ch4->DataType(ChannelDataType::FloatBe);
    ch4->DataBytes(8);

    writer->PreTrigTime(0);
    writer->InitMeasurement();

    auto tick_time = TimeStampToNs();
    writer->StartMeasurement(tick_time);

    for (size_t sample = 0; sample < 100; ++sample) {
      auto value = static_cast<double>(sample) + 0.23;
      ch1->SetChannelValue(value);
      ch2->SetChannelValue(value);
      ch3->SetChannelValue(value);
      ch4->SetChannelValue(value);

      writer->SaveSample(*group1, tick_time);
      writer->SaveSample(*group2, tick_time);
      tick_time += 1'000'000'000;
    }
    writer->StopMeasurement(tick_time);
    writer->FinalizeMeasurement();
  }

  MdfReader reader(mdf_file.string());
  ChannelObserverList observer_list;

  ASSERT_TRUE(reader.IsOk());
  ASSERT_TRUE(reader.ReadEverythingButData());
  const auto* file1 = reader.GetFile();
  const auto* header1 = file1->Header();
  const auto dg_list = header1->DataGroups();
  EXPECT_EQ(dg_list.size(), 2);

  for (auto* dg4 : dg_list) {
    const auto cg_list = dg4->ChannelGroups();
    EXPECT_EQ(cg_list.size(), 2);
    for (auto* cg4 : cg_list) {
      CreateChannelObserverForChannelGroup(*dg4, *cg4, observer_list);
    }
    reader.ReadData(*dg4);
  }
  reader.Close();

  for (auto& observer : observer_list) {
    ASSERT_TRUE(observer);
    ASSERT_EQ(observer->NofSamples(), 100);
    for (size_t sample = 0; sample < 100; ++sample) {
      double channel_value = 0;
      const auto valid = observer->GetChannelValue(sample, channel_value);
      EXPECT_TRUE(valid);
      EXPECT_FLOAT_EQ(channel_value, static_cast<double>(sample) + 0.23)
          << observer->Name();
    }
  }
}

[chrisakira]

It Worked!!! Thank you soo much for your help and quick response!!! Yeah i'm building a sorta of can logger here that writes data to MF4 and this lib is saving me

[ihedvall]

There exist an MDF bus logging standard. It require little bit of change of the current writer object as it must handle variable length records. Please let me know if you want that because it is in the pipe-line.

[chrisakira]

I would be very grateful if possible!!

[ihedvall]

When logging CAN buses, the input data is data frames (0..8 bytes more if CAN FD or XCP). Instead of logging channel values only the data frames are stored in the file. Normally is these loggers implemented in small embedded device iwth very little storage and computing power. CSS Electronics is a typical example. It's easy to configure as you only need to define which CAN ID you want to log. The draw-back comes when you want to read the file, it will return just the raw bytes. So you need a CAN DBC file.

They are loggers that do store both signals and raw bytes.

We need to agree of some type of use case/requirement. A meeting (MS Teams?) might be useful.

[chrisakira]

The project I'm working on would involve the second scenario, recording decode values and raw bytes, and it would be used for both J1939 protocol and XCP protocol (Possibly many more on the future). I've just started studying the MF4 standard, and if possible, I would like to schedule a meeting.

[ihedvall]

Yes. It sounds like a bigger project. The J1939 is easy but the XCP protocols might require some extra work to finalize as the ECU A2L file is involved. You can set-up a meeting. My email address is ihedvall@telia.com

[ihedvall]

I'm about to start the MDF bus logger writer. It's good timing to add your requirement to the project. Simplxs has left some comments. Starting up a A2L/XCP project as well but A2L/XCP is a rather large project.