Dealing with negative timestamps - MF4Writer Notifier

[henrique-rsilva]

DESCRIPTION

I'm using a notifier to read messages from my device CAN Bus, this notifier is equipped with the MF4 Writer listener. See below:

mf4_builder = can.MF4Writer(  # type: ignore
      file=Path(working_dir, f"mf4/can_listener_{log_filename}.mf4"),
      database=dbc_path,
      mode="wb",
      compression_level=COMPRESSION_LEVEL,
  )

  listeners: List[can.notifier.MessageRecipient] = [
      mf4_builder,
  ]

  can_bus.flush_tx_buffer()
  global notifier
  notifier = can.Notifier(can_bus, listeners)  # type: ignore

Everything all right until here.

To test my script I'm using a VECTOR 1630A + SW CANalyzer to generate some messages in my CAN bus.

Then, I start my CANalyzer simulation and my python script (notifier).

After my MF4 file reaches a specific file size, I close this file and start a new one, this is necessary to save RAM memory, I can't keep a large file size because I have a limited HW.

Below, the part of my code where I close the notifier, and so, also the MF4 file.

if flag == STATE["RUNNING"] and est_size >= FILE_SIZE:
    logger.info("Completed file size: Stopping current notifier ...")
    notifier.stop(timeout=5)
    logger.info("Notifier stopped.")
    flag = STATE["START"]

No problems until here.

BUT, this time between I close the current file (notifier) and open a new one, is enough to accumulate some messages in the CAN bus.

And here stands the problem, when the new file starts it consumes the messages accumulated in the CAN bus (before the new notifier start), and these messages are wrote in the MF4 file with negative timestamps (which make sense but generate a problem, because I don't want negative timestamps.)

See in the link below (google drive):

Negative Timestamps

That's my problem.

SOLUTION

To solve that I had to do a little change in the python-can library, in the mf4.py sheet.

The library timestamp adjustment occurs here:

 channel = channel2int(msg.channel)
 timestamp = msg.timestamp
 if timestamp is None:
     timestamp = self.last_timestamp
 else:
     self.last_timestamp = max(self.last_timestamp, timestamp)
 timestamp -= self._start_time  # timestamp = timestamp - self._start_time

Now, I'm doing this:

channel = channel2int(msg.channel)
timestamp = msg.timestamp
if timestamp is None:
    timestamp = self.last_timestamp
else:
    self.last_timestamp = max(self.last_timestamp, timestamp)

timestamp -= self._start_time  # timestamp = timestamp - self._start_time

# Henrique R Silva 2023-07-04 - Start
if abs(self.timestamp_factor) == 0.0:
    self.timestamp_factor = abs(timestamp)
    timestamp = 0.0
else:
    timestamp += self.timestamp_factor
# Henrique R Silva 2023-07-04 - End

QUESTION

Could someone of you, please, give me a feedback on this ?

Is this the way to solve that ?

Are there any other more elegant way to do it ?

And besides that, these negative timestamps, did you see some behavior like that before ?

I would appreciate any help that I could get here.

Thank you.

Additional context

OS and version: Debian 10 Python version: 3.7 python-can version: 4.2.2 python-can interface/s (if applicable): CAN1

[pierreluctg]

You probably want to open only a single Notifier and add/remove (and stop) listeners.

[henrique-rsilva]

Hello @pierreluctg, thank you for your suggestion.

I tested it. It worked.

See a snippet of my code:

if flag == STATE["RUNNING"] and est_size >= FILE_SIZE:
    logger.info("Completed file size: Stopping current listener ...")
    notifier.remove_listener(mf4_builder)
    mf4_builder.stop()
    logger.info("Listener stopped.")
    flag = STATE["RENEW"]

Snapshot of my achievement with listener solution

Listener Solution for Negative Timestamps

Question / Issue

Here in this point I have a question, let's see:

When I stop the listener and I start it again, it seems there are some data loss, I'll explain:

While the my script closes a MF4 file and starts another one, there are data running in the CAN bus. What about these data ? I need to collect these data as well.

I really can't lose any data.

Lookint at the ID number zero in the MF4 file (see the screenshot above) it's possible to see the evidence that some data were not collected, because the timestamp starts at almost in 1 second. And look, I haven't even mentioned all data flowing on the CAN bus in those 2-3 seconds between the closure and the beginning of a MF4 file (using listener add/remove).

When I was collecting data using the notifier (stopping and starting), somehow, it seems that the notifier could collect the data that were in the CAN bus even before it starts, that's why the negative timestamps. At least it's my understanding of that.

Could you give me some explanation on that ?

And besides, could you give some help to solve this in the best way ? I mean, with no data loss and keeping the positive timestamps.

I would appreciate very much your help.

Thank you sir.

[danielhrisca]

BUT, this time between I close the current file (notifier) and open a new one, is enough to accumulate some messages in the CAN bus.

when you build the next notifier you need to adjust some attributes

if flag == STATE["RUNNING"] and est_size >= FILE_SIZE:
    logger.info("Completed file size: Stopping current listener ...")
    notifier.remove_listener(mf4_builder)
    mf4_builder.stop()
    logger.info("Listener stopped.")
    flag = STATE["RENEW"]

    start_time = mf4_builder._mdf.header.start_time
    start_timestamp = mf4_builder._start_time
    last_timestamp = mf4_builder.last_timestamp

    mf4_builder = can.MF4Writer(  # type: ignore
        file=Path(working_dir, f"mf4/can_listener_{log_filename}.mf4"),
        database=dbc_path,
        mode="wb",
        compression_level=COMPRESSION_LEVEL,
    )
    mf4_builder._mdf.header.start_time = start_time
    mf4_builder.start_timestamp = start_timestamp
    mf4_builder.last_timestamp = last_timestamp