vsergeev
commented
7 years ago Hi Rhodey,
Awesome! I'm glad you like it. I'd be happy to collaborate on any features you're interested in and on the overall direction of the project.
Asynchronous control messages is the most requested feature by email, and is definitely a blocker for those trying to implement interactive flow graphs. Despite being in the long-term section in the wiki, its implementation isn't too difficult. It's mostly a matter of settling on a good first API, as it would be a very public interface. So this is a great opportunity to talk about it.
I also think it's time to start implementing it experimentally. Below is what I've had in mind so far. Ideas and feedback are always welcome.
High-level
- Asynchronous control in Lua
- Asynchronous control via RPC
- JSON-based protocol similar to JSON-RPC 2.0
- Built in RPC server with support for multiple transport layers
- UNIX, TCP; perhaps nanomsg, HTTP in the future
- Block-side
- Automatic registration of control handlers
- Control handlers run in block process as block methods
- Natural error propagation from control handlers with
error() - Minimal disruption to block's data path read-process-write loop
- Flowgraph-side
- Exporting Lua block instances to RPC with an instance name
- Control methods are namespaced under the instance name
- Meta-API for top-level blocks supporting
start(),status(),stop(), etc. - A new method
listen()for top-level blocks to run the built-in RPC server - Future: meta-API for flow graph block discovery and possibly method discovery
- Exporting Lua block instances to RPC with an instance name
Example
This is just a sketch up of what it could be like.
Example flow graph:
-- rtlsdr_am_envelope.lua example
local source = radio.RtlSdrSource(130e6, 1102500)
local decimator = radio.DecimatorBlock(5e3, 50)
local am_demod = radio.ComplexMagnitudeBlock()
local dcr_filter = radio.SinglepoleHighpassFilterBlock(100)
local af_filter = radio.LowpassFilterBlock(128, 5e3)
local af_gain = radio.AGCBlock('slow')
local sink = radio.PulseAudioSink(1)
local top = radio.CompositeBlock()
top:connect(source, tuner, am_demod, dcr_filter, af_filter, af_gain, sink)
top:start()After the flow graph is started, the programmatic interface allows calls into blocks from the flow graph process:
-- Asynchronous control in Lua
source:ctrl_set_frequency(135e6)
af_filter:ctrl_set_bandwidth(3e3)
af_gain:ctrl_set_mode('fast')These methods are monkey-patched after the top-level block is started to wrap the underlying interprocess calls.
Specific blocks can be exported to RPC under instance names:
-- Export control of these blocks to RPC, under the specified instance names
top:export(source, "rtlsdr")
top:export(af_filter, "af_filter")
top:export(af_gain, "af_gain")and the built-in RPC server can be run with listen(), accepting a URI specifying the transport:
-- Listen for asynchronous control over RPC (blocking)
top:listen("tcp://0.0.0.0:9000")The RPC speaks JSON and can be made by your language/transport of choice:
-- Block control methods namespaced under block instance
-> {"version": 1, "method": "rtlsdr.ctrl_set_frequency", "params": [135000000], "id": 1}
<- {"version": 1, "result": null, "id": 1}
-> {"version": 1, "method": "rtlsdr.ctrl_get_frequency", "id": 2}
<- {"version": 1, "result": 135000000.0, "id": 2}
-> {"version": 1, "method": "rtlsdr.ctrl_set_frequency", "params": ["foobar"], "id": 3}
<- {"version": 1, "error": "Invalid frequency.", "id": 3}
-> {"version": 1, "method": "af_filter.ctrl_set_bandwidth", "params": [2500.0], "id": 4}
<- {"version": 1, "result": null, "id": 4}
-- Top-level methods are top-level block methods
-> {"version": 1, "method": "status", "id": 5}
<- {"version": 1, "result": {"running": true}, "id": 5}
-> {"version": 1, "method": "stop", "id": 6}
<- {"version": 1, "result": null, "id": 6}
-> {"version": 1, "method": "status", "id": 7}
<- {"version": 1, "result": {"running": false}, "id": 7}Blocks control handlers are automatically registered by the prefix ctrl_ and are ordinary Lua methods that can access block state naturally:
local RtlSdrSource = block.factory("RtlSdrSource")
...
function RtlSdrSource:ctrl_set_frequency(frequency)
assert(type(frequency) == "number", "Invalid frequency.")
local ret = librtlsdr.rtlsdr_set_center_freq(self.dev[0], frequency)
if ret ~= 0 then
error("rtlsdr_set_center_freq(): " .. tostring(ret))
end
end
function RtlSdrSource:ctrl_get_frequency()
local frequency = librtlsdr.rtlsdr_get_center_freq(self.dev[0])
if ret ~= 0 then
error("rtlsdr_get_center_freq(): " .. tostring(ret))
end
return tonumber(frequency)
endComposite blocks can implement control handlers too, calling underlying blocks and even nested composite blocks, as needed:
-- AM Envelope Example
local source = radio.RtlSdrSource(130e6, 1102500)
local decimator = radio.DecimatorBlock(5e3, 50)
local am_demod = radio.ComplexMagnitudeBlock()
local dcr_filter = radio.SinglepoleHighpassFilterBlock(100)
local af_filter = radio.LowpassFilterBlock(128, 5e3)
local af_gain = radio.AGCBlock('slow')
local sink = radio.PulseAudioSink(1)
local top = radio.CompositeBlock()
top:connect(source, tuner, am_demod, dcr_filter, af_filter, af_gain, sink)
-- Custom control handler for top-level composite block
function top:ctrl_reset()
-- Reset to known state
source:ctrl_set_frequency(130e6)
af_filter:ctrl_set_bandwidth(5e3)
af_gain:ctrl_set_mode('slow')
end
function top:ctrl_foo()
return {1, 2, 3}
end
function top:ctrl_bar()
error('bar')
end
top:start()-- RPC to top-level methods
-> {"version": 1, "method": "ctrl_reset", "id": 7}
<- {"version": 1, "result": null, "id": 7}
-> {"version": 1, "method": "ctrl_foo", "id": 8}
<- {"version": 1, "result": [1, 2, 3], "id": 8}
-> {"version": 1, "method": "ctrl_bar", "id": 9}
<- {"version": 1, "error": "bar", "id": 9}Performance and Other Details
The control path would probably be implemented with anonymous UNIX sockets, like the data path. The only changes to the critical path of the block is adding a poll() or select() to monitor both control and data sockets for new data. This isn't too new -- blocks with multiple inputs already poll() across multiple data path sockets before reading -- but it does add an additional context switch to the critical path of all single input blocks (which currently just block on read()).
The JSON RPC request decoding, execution, and response encoding would be handled in the control path of blocks. The built-in listen() RPC server in the top-level block is essentially just a reverse proxy (over TCP, etc.) to the underlying control paths (over UNIX sockets) of its blocks. It receives a request from the client, looks up the right block by the instance name, strips the instance name from the call, forwards the request to the block, and then forwards the response from the block back to the client.
Outstanding Issues/Questions
- Should control handler arguments be statically typed?
- Pro: Allows for automatic type checking
- Pro: Allows for control handler API discovery
- Con: Nested or optional types are difficult to specify without an IDL
- Con: Adds complexity to registering a handler
I'm leaning towards making them purely dynamic types for now, as this feels more Lua and makes registration very easy. Control handlers can assert types as needed, much as block constructors do now.
In the future, control handlers could associate an API help text that is accessible through the meta-API on the top-level flow graph block. It's also possible to add type annotations to control handlers later, and leave the simpler dynamic typing for backwards compatibility, if needed.
I'd love to hear what you think!
rhodey
manfredhaertel
konsumer
Hi @vsergeev,
I really love luaradio and am eager to switch many of my DSP projects over to your project, the main thing preventing me from doing this right now is your long-term "Add API for asynchronous control messages" TODO. Have you made any progress on this recently? Is there a design plan you could lay out that I could maybe get started on?
Lua and especially Lua FFI are not my specialties but I'm highly motivated to migrate to luaradio. I've been working on the Radio Witness Project for about two years now and If I can migrate to luaradio I would be eager to contribute more.
Best,
-- Rhodey