Issues migrating an app using the Event API to periodically send heartbeats (from SDK3 to SDK4)

Please do not disclose security vulnerabilities as issues. See our security policy for responsible disclosures.

Describe the bug

The app is quite simple, and was initially develop from various examples provided by Axis for SDK3. Every 2 minutes, it will collect some metrics from the camera and emit an event (which will be published to MQTT). Now that SDK3 is becoming deprecated, I am migrating this app to the new SDK4. However, both versions of the app do not behave the same, to the point the version using SDK4 can be declared as broken.

When compiled with the legacy SDK3 and using ax_event_new() from the Event API

everything works as expected. In fact, this code has been running for several years on several hundreds of cameras.

2024-07-16T14:19:45.640+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2580354]: Send heartbeat
    ... about 120s later, we send another heartbeat, as expected
2024-07-16T14:21:56.890+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2580354]: Send heartbeat
    ... about 120s later, we send another heartbeat, as expected
2024-07-16T14:24:07.891+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2580354]: Send heartbeat
    ... about 120s later, we send another heartbeat, as expected
2024-07-16T14:26:18.890+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2580354]: Send heartbeat

When compiled with the new SDK4 and using the non-deprecated ax_event_new2() from the Event API

The behavior is more chaotic, though very systematic

2024-07-16T13:50:18.390+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2575994]: Send heartbeat
    ... only about 6 seconds (expected about 120s)
2024-07-16T13:50:24.640+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2575994]: Send heartbeat
2024-07-16T13:50:26.576+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2576826]: Heartbeat started
    ... heartbeat app rebooted
    ... waited 6 minutes before heartbeat was sent (expected either right after start, or about 120s later)
2024-07-16T13:56:49.390+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2576826]: Send heartbeat
    ... again, only about 6 seconds (expected about 120s)
2024-07-16T13:56:56.140+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2576826]: Send heartbeat
2024-07-16T13:56:58.140+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2577663]: Heartbeat started
    ... again, heartbeat app rebooted
2024-07-16T14:03:20.640+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2577663]: Send heartbeat
    ... again, only about 6s (expected about 120s)
2024-07-16T14:03:26.389+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2577663]: Send heartbeat
2024-07-16T14:03:28.890+02:00 axis-b8a44fafc67e [ INFO    ] heartbeat[2578441]: Heartbeat started
    ... again, heartbeat app rebooted

The two version of the app use the very same code. The only difference is isolated in a function where we use conditional compilation directives, like this:

static AXEvent* new_event(AXEventKeyValueSet *key_value_set) {
  #if AXISSDK == AXISSDK4
  return ax_event_new2(key_value_set, NULL);
  #else
  return ax_event_new(key_value_set, NULL);
  #endif
}

To reproduce

A this point, I do not have a minimal reproduceable setup that I can share.

Before I invest time on making this reproduceable setup, I would appreciate if you have any feedback, hints, etc that could help me fixing the issue with SDK4. Are there any more changes in the Event API that could explain the issue?

Screenshots

None. Logs have been provided

Environment

Axis device model: M1075-L (though the issue is probably not limited to this model)
Axis device firmware version: 11.10.83 (same comment)
Stack trace or logs: logs attached previously
OS and version: Irrelevant, binaries are compiled with the docker containers provided by Axis
Version: SDK4

Additional context

Add any other context about the problem here.

AxisCommunications / acap-native-sdk-examples