Extended RSS Implementation

[bvacaliuc]

In #11, we provided a basic integration with Jim Sky's Radio Sky Spectrograph. There is a desire to extend the spectrum output to include more frequency bins and to use full precision. Essentially to produce rtl-power-compatible output files.

Here is the relevant discussion:

On Thu, Aug 4, 2016 at 7:16 AM, Bogdan Vacaliuc bvacaliuc@ieee.org wrote:

re: the connection string for the extended format output. How do you want to do this? Keep the same 4 key,value pairs as RSS and add more key, value using the same parsing format? Can we use keys with more than just one character in the extensions if so? Switch to .json?

re: the format for the scans. should we use a binary version of the rtl-power output? How about this variation:

timestamp (IEEE-754 8-byte double) hz_low (IEEE-754 4-byte float) hz_high (IEEE-754 4-byte float) hz_step (IEEE-754 4-byte float) samples (4-byte integer) db-values (IEEE-754 4-byte float) * CHANNELS (as given in the header)

I dont think we need the end code marker. I would have preferred to have a stateless stream, but the method of disconnecting when offset/center frequency and channels are changed seems to work very well.

On Thu, Aug 4, 2016 at 11:18 PM, Tony Bigbee tony.bigbee@gmail.com wrote:

Re: key value pairs. I suggest extending the key to a full variable name and just separating by space with the value and keeping the pipe delimiters. I would also be fine with JSON --it's easy to parse in any language.

Ditching the end code marker is perfectly fine with me. I can easily dynamically compute the scan boundaries.

Here is the detail on the "Extended RSS" implementation simiar to Jim's spec for RSS:

The RASDRproc program acts as a TCP server and listens for a connection on the port defined by the 'Setup RSS Output' as described here. Configure your program to connect to this IP:PORT combination. Upon a connection, RASDRproc accepts the connection and sends an 1024-byte ASCII string as a response. The response will be key value combinations separated by the "|" character.

The following keys will be provided:

• [CenterFrequencyHertz] The tuned center frequency in Hz
• [BandwidthHertz] The bandwidth of the observation in Hz
• [OffsetHertz] The offset frequency (in case an LNB or up/down converter is used with the receiver)
• [NumberOfChannels] The number of frequency bins that will be provided for each scan line

The following keys may be provided:

• [IntegrationTimeSec] The amount of integration in seconds that will be used for each scan line
• [GainDb] The total amount of receiver gain used
• [NotesString] An ASCII string provided by the program to describe the observation (excluding the "|" character).

For example: CenterFrequencyHertz 21000000|BandwidthHertz 5000000|OffsetHertz 0|NumberOfChannels 2048|

This would tell your program that the center frequency of the observation is 21 MHz. The bandwidth is 5 MHz, so the display will run from 21 + 2.5 to 21 - 2.5 MHz, or 18.5 MHz to 23.5 MHz. There is no offset so no frequency converter is used. Finally, NumberOfChannels denotes the number of channels that will be reported for each of the spectrum scans. There is no predefined limit on the number of channels, but the number is limited to a 4-byte unsigned integer. The connection block will be a fixed-size of 1024 bytes. Each of these parameters is separated by the pipe symbol "|". The parameter string will end with a CRLF. There may be trailing NUL to complete the 1024-byte connection block description.

Following this, a constant stream of binary data will be provided, the total size which is known after the connection string is parsed. It is not necessary to interpret the connection block (above) as the binary stream will self-describe itself. It will consist of the following binary data elements (equivalent to a binary-version of the rtl-power specification) in Network Byte Order:

• [timestamp] IEEE-754 8-byte double representing the seconds since January 1, 1970 (the Unix Epoch)
• [hz_low] IEEE-754 4-byte float representing the center frequency of the lowest frequency bin
• [hz_high] IEEE-754 4-byte float representing the center frequency of the highest frequency bin
• [hz_step] IEEE-754 4-byte float representing the difference between two adjacent frequency bins
• [samples] A 4-byte unsigned integer representing the number of samples used to integrate each frequency bin
• [channels] A 4-byte unsigned integer representing the number of frequency bins to follow
• [db-values] IEEE-754 4-byte float * channels (as given in the header above)

A useful set of links for timestamps is given here

[bvacaliuc]

With commit 65eb88cc0a65aed93413d19f7ac18c0787c4ea76, I have updated the test code to parse both the RSS and extended RSS packets.

I chose to make the packet headers self-describe themselves, such that the ASCII connection string contains current metadata and is informative but not strictly necessary for the proper interpretation of the spectrogram stream. I also chose to make the connection string be a fixed-size ASCII data block to avoid dealing with alignment issues. I'm not completely happy with the key-value notation currently in use, but I didn't want to spend the effort integrating a JSON generator at this time. Regarding that, and looking at this json library analysis, I think that the project is best served by using sheredom's json.h which can be added as an external module to the dependencies folder of RASDRproc.

I chose to use Network Byte Order for the binary stream, as noted in the literature.

[bvacaliuc]

On Wed, Jan 11, 2017 at 11:26 PM, Tony Bigbee tony.bigbee@gmail.com wrote:

Bogdan - Per the attached screenshot, the extended RSS is reporting frequency amplitudes in what I think is reverse order. I' just print the time stamp and psd data structure every time step. Notice the increasing amplitude from low to high frequency in the graph, and the reverse in the psd.

I am able to confirm that this is a problem, performing a capture and using the 'heatmap.py' from Kyle Keen's page:

What appears to have happened is that the order of frequencies to support the RSS (#11, highest frequency to lowest frequency) is being emitted for both RSS and RSSx codes, rather than the correct order for each case (RSSx should emit lowest frequence to highest frequecy). The fix requires an update to both RASDRproc code and the rss2rtl-power script to handle the ordering correctly in each case.