Fix - Race condition when trying to add a log file that's been scheduled for removal

Fixes https://sentinelone.atlassian.net/browse/DTIN-4631
__lock has been change to reentrant RLock.

Based on the comments the lock was initially meant to lock status fields used in generate_status. Now it ensures synchronized access to fields needed for asynchonous workflow of log files. This is already causing a timeout when retrieving status from the agent (https://sentinelone.atlassian.net/browse/DTIN-4769) and will be handled in another PR.

The Bug:

When a log file (path, worker_id) is scheduled for removal, the removal is done in two separate steps:

__remove_logs_scheduled_for_deletion
- Instructs all matchers from __logs_pending_removal to finish
- Clears __logs_pending_removal
__purge_finished_log_matchers
- Removes all finished matchers from __logs_pending_removal, __dynamic_matchers, __dynamic_paths

The race condition can be reached when a path is scheduled for removal and the matchers are still finishing. When add_log_config is called it considers log path to be active and skips adding it. The log file is later removed.

In log we can see sequence like this one:

Log ( log_path='xy', worker_id='default' ) for monitor 'scalyr_agent.builtin_monitors.kubernetes_monitor' is pending removal

Tried to add new log file 'xy' for monitor 'scalyr_agent.builtin_monitors.kubernetes_monitor', but it is already being monitored by 'scalyr_agent.builtin_monitors.kubernetes_monitor'

Removing log file ( log_path='xy', worker_id='default' ) for 'scheduled-deletion'

scalyr / scalyr-agent-2

Fix - Race condition when trying to add a log file that's been scheduled for removal #1302

Test Results