Exception Data for Streaming LTE sensors.

[ranganathanm]

In private emails we have discussed the following architecture:

• Streaming sensors will monitor only one band.
• The server will accumulate ALL data streamed by a streaming sensor as "occupancy data".
• Any data POSTED by a streaming sensor will be logged as "Exception Data" and is meant for "spectrum policing"
• Exception data will not be used for occupancy computations.
• Exception data can have different types
  • DataMessage type is a spectrogram (this has been implemented by Tim)
  • I/Q data is a pointer to data stored on the sensor and can be sent to the server as a URL (later accessed for spectrum policing or may be sent by sensor asynchronously).

This discussion thread will discuss how to deal with Exception Data. Please discuss the following:

• Does this architecture make sense?
• What kinds of data should be stored on exception?
• How are exceptions generated ( autonomously by the sensor or on trigger from the server ).
• Should exception events be displayed from the user display ( may not need login ) or from the admin interface?
  • How should exception data be visualized.

[msouryal]

Regarding how exceptions are generated, autonomously by the sensor or triggered from the server, I can think of reasons for both. On one hand, the server might trigger an exception capture when coarse occupancy exceeds a user-defined threshold at the server. On the other hand, the sensor might trigger an exception capture when it detects a certain type of signal. Currently, the sensor is a basic (frequency-domain) energy detector. But in the future, it could run more sophisticated algorithms to detect specific signals, like a radar pulse or the LTE downlink synchronization signal. The sensor might trigger an exception when, for example, it detects co-channel radar and LTE.

Question on using a URL pointer to I/Q data at the sensor: Would that create a security risk at the sensor, having the sensor open up https ports?

[MichaelCotton]

Warning: This is a bit roundabout... These comments are related to the radar sensors that are being deployed this year and next. I can see exception data and triggering being useful. In addition to MSOD/sensor triggered exception data, it would be great to have a workflow capability at MSOD where emails w info are sent to specified users under certain conditions.

I know the topic at hand is streaming and there is good reason for that dev, but I see that as being a little down the road for us, i.e., at the time when the lowest possible latency is required and dedicated sensors for a given band and System2Detect are required.

This year, we are putting out sensors that have the hardware and software capability to measure in three bands. Right now, we can only schedule measurements in one band. However, once the FY15 installations are complete, our next release of the sensor firmware will allow for scheduled measurements. An example schedule that we will likely use in FY16 is: (1) 3.5 GHz SPN43 - as frequently as schedule/detection allows, (2) 3.0 GHz BoatNav radar - as frequently as schedule/detection allows, and (3) 2.8 GHz Airport Surveillance radar (ASR) - hourly. We also plan to implement a 3.5 GHz ~LTE detection scheme. In FY17, a likely measurement schedule will be: (1) 3.5 GHz SPN43 - as frequently as schedule/detection allows and (2) 3.5 GHz ~LTE - as frequently as schedule/detection allows. We will see if people want us to continue monitoring the 2.8 GHz and 3.0 GHz bands. I am thinking not, bc utilization in those bands is pretty static.

It seems as though we will be posting data bc we are scheduling more than one band or more than one Sys2Detect. Exception data is data acquired outside of some norm. For the radar sensors, that norm is defined by the schedule. Before taking the exception discussion further I want to ask - Do we want to control sensor schedules (to measure multiple bands and multiple Sys2Detect) from MSOD? If so, perhaps an MSOD API get command should be developed to allow for the sensor to query MSOD for a schedule after each data post.

[MichaelCotton]

To address Michael's question about IQ data at the sensor... the feedback that I am getting is that IT Security level is heightened if we state that IQ data is handled anywhere in the network. I think we should avoid it until someone asks for it specifically.

[djanderson]

Going to be a bit pedantic here.

I don't think "Exception" is the right word to describe the data that's going to be sent over this channel. Might "Event" be a more appropriate term? Exceptions should be, well, exceptional:

• an LTE sensor identifies an LTE signal: that's a noteworthy event and should be logged
• an LTE sensor's self-check routine fails to detect its signal source: that's an exception... definitely outside of how the sensor is supposed to be operating

What kinds of data should be stored on exception?

• timestamp
• sensor ID
• event "category" to help route the message properly
• event description string (can standardize description string for easy parsing on a per-sensor-type basis, ala syslog. I don't think finding a combination of JSON fields to fit every sensor/event combination is necessary)

"Exception" could be one of the event categories so that truly exceptional events are handled appropriately.

How are exceptions generated ( autonomously by the sensor or on trigger from the server ).

• I would definitely vote for autonomously by the sensor, especially considering time sensitive applications like policing. The sensor encounters a notable event, it formats the event message and POSTs it to the server, retrying until it gets a "success" ack back from the server.

[ranganathanm]

Catching up... I agree that "Exception" is a misnomer ("Extraordinary" would be a better word to use but has an unnecessarily dramatic connotation. )

I like the scheme proposed by Doug i.e. "Event", with "Exception" being one category and "Error" being another category. It makes this general.

I had not meant "Exception" in the sense of being an Error. Error logging is an important aspect and it should also be handled by posting a message to the server but we should treat it separately.

In response to Mike's question.

Do we want to control sensor schedules (to measure multiple bands and multiple Sys2Detect) from MSOD? If so, perhaps an MSOD API get command should be developed to allow for the sensor to query MSOD for a schedule after each data post.

This would be part of the sensor configuration. In fact all the measurement parameters also need to be added to the sensor configuration. The sensor would read this configuration before it starts to function. The data message effectively contains the configuration that is read by the sensor so this information would be sent back on each data message. If the configuration does not match, the server would reject the post and the sensor goes back and reads the configuration again, reconfigures itself and starts sending data again. This is less messaging than the sensor reading the configuration each time before POSTing and it avoids race conditions.

I think we need to defer on this topic until the next release of MSOD as there appear to be a number of things to consider.

[ranganathanm]

with respect to Mike's comment:

.. the feedback that I am getting is that IT Security level is heightened if we state that IQ data is handled anywhere in the network. I think we should avoid it until someone asks for it specifically.

We would expect to store I/Q for LTE data. Given that this is not of a sensitive nature and furthermore, given that it is encrypted (per the LTE standard), I am not sure why there is an IT Security concern. We would need I/Q for spectrum policing (presumably for our next release?)