Adding a Plugin System

[robotastic]

I am setting up this Issue to track things as I am building out this feature. It is being done in the plugin branch: https://github.com/robotastic/trunk-recorder/tree/plugin

[robotastic]

One thing to think through is how to handle deleting recordings and JSON files in the call_end hook. The Call Uploading happens on a separate Thread right now. That seems like the right move because you don't want the main TR thread to block for the Network to do things. Right now, without the plugin, the End_Call() function will handling deleting the files, unless an uploader is defined. In that case it lets the Call Uploader handle the deletion. This happens inside the thread, after all of the upload has occurred.

However, if we have multiple call_end plugins, how can you know when it is safe to delete the files, especially if some of the plugins are using threads.

One option could be to make all of call_end in plugman be a thread and none of the plugins are threads.... and then after all of the files are done, handle deleting things. You would have to copy all of the correct Call and Config info into a new struct and pass it to the thread.

Another option is just to not delete the files in trunk recorder. That could be handled by a daily chron job...

Overall, having a plugin be in charge of deleting files seems like a bad idea... also having the deletion happen in the plugin manager seems weird too, or atlest not the clearest approach

[robotastic]

You know - the deletion question could already be a problem if you are using both an upload script and the call_uploader. If the call_uploader finishes before the script, it will delete the files. So... this sort of suggests this is already the wrong way to handle file deletion.

https://github.com/robotastic/trunk-recorder/blob/09a2fbf25e2d18691f561d314b964be94254b773/trunk-recorder/call.cc#L177-L187

[robotastic]

Hmm... also - how should you handle compressing the audio file. Ideally this should be a single time and then be made available to the Plugins. Again though, this should be done in a thread. This is also sort of an argument that all plugin calls should be Threads.

[leee]

(Just butting in here, hope you don't mind):

Another option is just to not delete the files in trunk recorder. That could be handled by a daily chron job...

This may be a bad idea for users with limited disk writes, or for users who have limited disk space (embedded disk or even ramdisks).

However, if we have multiple call_end plugins, how can you know when it is safe to delete the files, especially if some of the plugins are using threads.

Hmm... also - how should you handle compressing the audio file. Ideally this should be a single time and then be made available to the Plugins. Again though, this should be done in a thread. This is also sort of an argument that all plugin calls should be Threads.

I think you'll want a single thread per call that handles the following in concurrency:

1. preprocessing (e.g. audio compression)
2. parallel jobs (e.g. call uploads to various platforms, inserting call data and metrics into databases, etc.)
3. postprocessing (e.g. uploadScript)
4. safe to delete

But there's also the matter of how to handle calls that fail any one of these steps or parallel jobs. What would best support a (queued?) retry mechanism?

[robotastic]

Not at all!

Yea - it looks like a single thread that runs through all the steps in order will be the cleanest approach. How about this:

when end_call() is called, it wraps up everything internally for the call. It shuts down the recorders and gets the final tally on length. It then puts all of the information about the call into a struct, like the call_info struct used in the uploader. Running on its own thread, there would be a call completion manager. end_call() would pass a filled out call_info struct over to the completion manager. After that, the Call object could safely be deleted in the main TR thread and removed from the vector.

The call completion manager would have a queue of call_info structs to go through. It would have a thread pool of workers. It would take a call_info from the queue and pass it to the worker, which would then go through the steps you outlined. If one of the steps fails, it could go back into the queue, with a expo-backoff timer on it. Each of the processing steps would have to be safe to do twice... or there would have to be a complex state/resumption mechanism. Some of this could be eased with a unique ID for each call. If the same call ID is uploaded twice, the online service could fail the second one.

An approach like this could be useful for delaying certain talkgroups by a set amount. Assuming the queue is large enough, you could just have some call_infos hang out there for 20 minutes before the manager dispatches them to a worker thread.

[robotastic]

I have the plugin system and improved call_concluder pretty much working as planned with this commit: 554d5f783b4904b6d356a0da0c606a7ecc1a189c It uses the Boost::Plugin approach, which makes it easy to use C++ classes in a dynamic library. All of the data for the plugin gets stored inside the class. There is then a vector of plugins to keep track of everything. For the call concluder, there is a structure that keeps track of all of the information for a call. It gets pass to a thread which will go through all of the steps to close out a call:

• conversion to m4a
• create a JSON
• run the upload scripts
• run the plugins if it fails on the plugins, it will be retried at a later date. There is a vector that keeps track of all the jobs it has to try in the future.

[robotastic]

I am currently working on rolling in the many_files branch: 97b32cfb1f227285ad4b085c6755e1485ce94877 I am planning on changing the entire way we look at Calls. Right now Calls are driven by messages on the control channel. A GRANT or UPDATE message reserves a frequency for a talkgroup. While messages keep coming through, it keeps that channel. When messages stop coming on the control channel, the call will end after a timeout period and all recording will stop. This generally works, but the problem is that some times a recorder can get backed up and voice frames are still waiting in its queue. There is a special frame called a terminator, that goes over the voice channel when a transmission is complete. While the Control Channel is good for starting a Call, when it finishes should be governed by activity on the voice channel and receiving the termination flag.

Proposed approach: A Call will represent a Freq being designated for a Talkgroup. This will set the Call's status to Active and assign a recorder to that frequency. A Call's status will remain Active while it:

• Receives GRANT & UPDATE messages on the control channel

If a Recorder receives a Termintor frame while its associated Call is in active mode, the current file will be closed and pushed into a Vector inside the call. The Recorder/wav_recorder will set its state to INACTIVE. A new Transmission/file will be started if the recorder/wav_recorder begins to receive new frames after that. The Recorder's status will change back to active.

A Call while change its status to INACTIVE when it stops receiving these messages on the control channel for a TIMEOUT period. When a Call becomes INACTIVE, it will still stick around. A Call will only end if:

• a Recorder receives a terminator frame while the Call is inactive OR
• Both the Call and Recorder are in an inactive state
• The call is in an INACTIVE state and the recorder is in an ACTIVE state, but its last activity is past a threshold (This is to target the case where the Terminator transmission may have been lost)

If a Call is on a Freq, but is INACTIVE, if a new GRANT/UPDATE is heard for the same Freq/TG, a new call will be created.

[robotastic]

Digging into things more, a couple observations:

• the wav_sink probably can not directly end a call. This is because it ends up being circular. The wav_sink would call end_call(), which when then stop the recorder, which would then close the wav_sink. With the mutex locks in place, this would deadlock things.

• instead, I think we need to use states and occasional cleaning. Here are the proposed states:

  • Active: the call has active GRANT/UPDATES, the wav_sink is actively recording something with an open file and has not hit a terminator yet
  • Idle: For a call, this only applies for analog/conventional P25 calls. This means that everything is still good to go, but Squelch hasn't been triggered yet. For a recorder, this means it is still associated with an active call but it either just hit a terminator or got a squelch EOB frame.
  • Inactive: Once an object hits the inactive state, it can not start going again without being closed out. Before a Call can be deleted, it has to enter into erasable state. For a recorder, it has to first stop(). When the wav_sink is in this state, it should not start any new transmissions.
  • Erasable: Only a call can enter into this state. It does this when:
    • The Call gets an end_call() and the associated recorder is already in an Idle state. In this case it will set the wav_sink to be inactive and block any new transmissions from being written. In this, it might be safe to just go ahead and conclude the call.
    • The wav_sink call add_transmission, meaning the current transmission is over, and the Call is in the Inactive state. In this case, the Call can not be directly be concluded. Instead it is set to erasable and then cleaned up on the next go around.

• To handle Unit to Unit calls, we may need to do something special. This is because it GRANT/UPDATE messages may not be continually sent out after call is established. It maybe enough to simply have the call stay open until a terminator flag shows up on the wav_sink.

[aaknitt]

will the uploadScript and unitScript functionality still remain once the plugin system is in place? Both are very useful in triggering external Python scripts based on different types of system activity. I built 4.0-beta and unitScript doesn't seem to be working...not sure if that's by design or just due to the beta nature.

[hayden-t]

This is there now in 4, but there seems to be more plugins than doc for, maybe plugins should mandatory have a readme with them in their folder.

[Dygear]

What about using the call_end hook like this ... Have it return either PLUGIN_CONTINUE (0) or PLUGIN_HANDLED (1).

If a plugin is not doing anything with that file, it simply returns PLUGIN_CONTINUE. If the plugin has handled the file, by for example deleting it, or maybe marking it as something to save for later it can return a PLUGIN_HANDLED value and in that case, plugin calls should break at this point. This does mean that it's possible that lower priority plugins will get starved of data because their functions are never called as a higher priority plugin has already handled that call. We are getting slightly more into @devicenull's territory here as this is an AMX Mod-ism.