usdot-fhwa-stol / cda-telematics

Apache License 2.0
1 stars 2 forks source link

NATs callback remains active in rosbag2_processing_service even when the rest of the service crashes #196

Closed adev4a closed 4 months ago

adev4a commented 6 months ago

Summary

The rosbag2_processing_service runs the NATS callback on a separate thread in order to allow the service to continue adding logs to the queue, even when a log file is being processed. Since processing of a rosbag happens one log file at a time, processing requests from the UI can still be accepted and handled using a queue on a first in first out basis.

The current implementation achieves this objective, by keeping the callback running independently of the rest of the process but causes an issue in an error case where a fault in the processing causes one part of the service to stop while keeping this callback still active.

This was tested by removing the required mysqldb connection in the middle of processing a rosbag. Since this connection is required, the unavailability causes part of the service to stop. However the nats callback remains active and the service still receives new requests from the UI. image

Commit

Develop

Expected Behavior

The rosbag2_processing_service attempts to restart after experiencing failure.

Actual Behavior

Nats callback remains active, even if the rest of the service has stopped working due to failure.

Steps to Reproduce the Actual Behavior

  1. Start rosbag2_processing_service with web-ui, mysql and influxdb connected
  2. Send request to process a rosbag from the UI
  3. Stop mysql container
  4. Send another process request from the UI

Related Work

No response