MITHaystack / digital_rf

Read, write, and interact with data in the Digital RF and Digital Metadata formats
Other
97 stars 30 forks source link

Digital RF Source block and handling missing values #4

Open ryanvolz opened 5 years ago

ryanvolz commented 5 years ago

This issue was originally raised in #3 here. Executive summary: The Digital RF Sink writes data in continuous mode by default, and when given 'now' as the starting write time this will almost always result in missing samples at the beginning of the first file, stored as NaN for floating point types. The Digital RF Source defaults to beginning to read the data at the start of the file, so it starts by outputting those NaNs. Other GNU Radio blocks subsequently choke on the NaNs and playback fails.

A detailed summary can be found in #3 here. The options for fixing this behaviour include:

ryanvolz commented 5 years ago

Commit 0f2fa4ae2527717cbccb0f9747305bc543da8aff, included in the 2.6.2 release, should partially solve this problem by outputting the missing values (NaN or minimum integer) as 0 when read from the Digital RF Source blocks in GNU Radio. At least now the playback should not fail, even if the user might see unexpected zeros.

I still think we need to do more to address this so the behavior is not as surprising. I recently tested write performance in chunked mode again, and there is still a significant speed penalty to defaulting to that. I don't think it makes sense to change chunked to the default.

Improving the documentation would go a long way I think, so we should do that before I would consider this issue resolved. There's still room for considering whether or not we want to further modify the Digital RF Source to skip over missing samples that are actually contained in the files with continuous mode (as opposed to just missing chunks in the files as in chunked mode).