search

MITHaystack / digital_rf

Read, write, and interact with data in the Digital RF and Digital Metadata formats
Other
97 stars 30 forks source link

'py_io_signature' object does not support indexing #3

Closed devnulling closed 5 years ago

devnulling commented 5 years ago

I'm trying to replay samples using the Digital RF Source that have been stored with the Digital RF Sink block and running into the error:

Traceback (most recent call last):
  File "/home/user/top_block.py", line 215, in <module>
    main()
  File "/home/user/top_block.py", line 203, in main
    tb = top_block_cls()
  File "/home/user/top_block.py", line 163, in __init__
    min_chunksize=None if 0==0 else 0,
  File "/usr/local/lib/python2.7/dist-packages/gr_digital_rf/digital_rf_source.py", line 602, in __init__
    out_sig_dtypes = [src.out_sig()[0] for src in self._channels]
TypeError: 'py_io_signature' object does not support indexing

/home/user/data structure:

data
├── ch0
│   ├── 2018-12-03T05-00-00
│   │   ├── rf@1543814115.000.h5
│   │   ├── rf@1543814116.000.h5
│   │   ├── rf@1543814117.000.h5
│   │   ├── rf@1543814118.000.h5
│   │   ├── rf@1543814119.000.h5
│   │   ├── rf@1543814120.000.h5
│   │   ├── rf@1543814121.000.h5
│   │   ├── rf@1543814122.000.h5
│   │   ├── rf@1543814123.000.h5
│   │   ├── rf@1543814124.000.h5
│   │   └── tmp.rf@1543814125.000.h5
│   ├── drf_properties.h5
│   └── metadata
│       ├── 2018-12-03T05-00-00
│       │   └── metadata@1543814115.h5
│       └── dmd_properties.h5
└── ch1
    ├── 2018-12-03T05-00-00
    │   ├── rf@1543814115.000.h5
    │   ├── rf@1543814116.000.h5
    │   ├── rf@1543814117.000.h5
    │   ├── rf@1543814118.000.h5
    │   ├── rf@1543814119.000.h5
    │   ├── rf@1543814120.000.h5
    │   ├── rf@1543814121.000.h5
    │   ├── rf@1543814122.000.h5
    │   ├── rf@1543814123.000.h5
    │   ├── rf@1543814124.000.h5
    │   └── tmp.rf@1543814125.000.h5
    ├── drf_properties.h5
    └── metadata
        ├── 2018-12-03T05-00-00
        │   └── metadata@1543814115.h5
        └── dmd_properties.h5

8 directories, 28 files

This is using GR 3.7.13.4, numpy 1.15.4

I've tried saving/replaying both ints and complex floats, coming up with the same error.

Example flowgraphs: usrp_drf drf_source

Any suggestions are appreciated.

ryanvolz commented 5 years ago

This looks like a compatibility problem with GNU Radio 3.7.12+ that was fixed in 7baede2, which is in the 2.6.1 release. Upgrading digital_rf should fix your problem. 🙂

devnulling commented 5 years ago

@ryanvolz thanks for the quick response!

I had found that commit since posting the issue and figured I'd try rolling back to GR 3.7.12.0 to see if it would resolve it with the version pip installs (ubuntu 16.04.5). After this build is complete, I'll try updating digital_rf.

ryanvolz commented 5 years ago

I think you'd actually need to go back to 3.7.11 for it to work, if I'm remembering correctly.

On Mon, Dec 3, 2018, 12:57 AM devnulling <notifications@github.com wrote:

@ryanvolz https://github.com/ryanvolz thanks for the quick response!

I had found that commit since posting the issue and figured I'd try rolling back to GR 3.7.12.0 to see if it would resolve it with the version pip installs (ubuntu 16.04.5). After this build is complete, I'll try updating digital_rf.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MITHaystack/digital_rf/issues/3#issuecomment-443597588, or mute the thread https://github.com/notifications/unsubscribe-auth/AEMBJ3qQeTj4oCo6nCFCI990g34oPA4Eks5u1L1LgaJpZM4Y96BV .

devnulling commented 5 years ago

It looks like pip is installing 2.6.1:

>>>drf.__version__
u'2.6.1'

I see the commit https://github.com/MITHaystack/digital_rf/commit/7baede22e5d32ac9fd3841a32d32cf15c0ae2274 changed the sink block, my error was being thrown on the Source.

I updated this line linked below to match the list() call, which gets past the error, but it doesn't produce samples

https://github.com/MITHaystack/digital_rf/blob/master/python/gr_digital_rf/digital_rf_source.py#L602

ryanvolz commented 5 years ago

Ah, my mistake, there is clearly still a problem with the source block. I can look into it in more detail tomorrow, push a fix, and then see about making a new release.

Does wrapping with list() make everything work, or are there still problems?

On Mon, Dec 3, 2018, 1:20 AM devnulling <notifications@github.com wrote:

It looks like pip is installing 2.6.1:

drf.version u'2.6.1'

I see the commit 7baede2 https://github.com/MITHaystack/digital_rf/commit/7baede22e5d32ac9fd3841a32d32cf15c0ae2274 changed the sink block, my error was being thrown on the Source.

I updated this line linked below to match the list() call, which gets past the error, but it produce samples

https://github.com/MITHaystack/digital_rf/blob/master/python/gr_digital_rf/digital_rf_source.py#L602

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MITHaystack/digital_rf/issues/3#issuecomment-443601149, or mute the thread https://github.com/notifications/unsubscribe-auth/AEMBJ0ggdr01HmgQjyM0nVloPVVpLSbhks5u1MLBgaJpZM4Y96BV .

devnulling commented 5 years ago

Wrapping with list() will get rid of the original error, but the source block will not stream samples. This is running on GR 3.7.12.0.

devnulling commented 5 years ago

Digging into this more, I've found the Digital RF Source block is actually streaming samples, however, it is not stripping off some of the header information which GR doesn't like.

Here's my testing so far:

1) Noise source into a DRF Sink and regular GR file sink one.fc32 drf_gen

2) Read the DRF generated files with the DRF Source block and saved it to a new file using the GR File Sink two.fc32 drf_convert

In two.fc32 there is a header of the bytes 00 00 C0 7F repeating, which GR won't process as a stream.

drf_hex

Removing the offending bytes (00 00 C0 7F , 4424 in this capture) from two.fc32 and saving it to a new file, GR will accept / stream from a File Source block.

It seems the header is not being parsed correctly by the Digital RF Source block.

one.fc32.gz two.fc32.gz rf@1543900016.000.h5.gz

drf_convert.grc.gz drf_gen.grc.gz

ryanvolz commented 5 years ago

Well, it looks like there are multiple factors at play here, and it's something I wouldn't anybody who's not familiar with the inner workings of Digital RF to anticipate.

First, Digital RF actually has two formats for writing files: continuous and chunked. In the default continuous mode, each file is a fixed size and stores a value for every sample that falls within its time window. If the write starts after the time where the file's time window starts, or if samples are skipped, it stores a default "missing" value. For floating point types, this is NaN. For integer types, this is the minimum (most negative) value. In chunked mode, missing samples are not included in the file, and each file can have multiple chunks of data. Continuous mode is the default because it's faster when reading the data to not have to glue together potential file chunks. We may need to revisit that, though.

Second, because you're using the default sink settings and setting the data start time using 'now', your test data is being written as 32-bit complex floats in continuous mode with a 1-second file cadence and the data is almost surely not starting at an integer second. So, the files that you're writing have some number of 'NaN' samples at the beginning.

Third, when the Digital RF Sink reads a channel, it starts by default at the beginning of the data present in the first file. So in this case, it's starting with some number of 'NaN' samples.

Fourth and finally, it appears that GNU Radio does not like 'NaN' values, so I guess it's refusing to move past them and then actually start streaming the real data.

Overall, this is a case where the documentation and default behavior are not clear enough about what's happening, and it leads to a non-functional setup. I haven't run into this before because we typically avoid it in one of two ways: 1) set a data start time that falls exactly on an integer second, so there are no "default" samples at the beginning of the first file in continuous mode, and 2) write data as an integer type, because this is usually what the radios provide at a base level (the default value for integer samples is still a valid integer, do GNU Radio does not choke on it).

The quickest fix for your setup is to change the Sink start time to 'nowish' from 'now', which rounds to an integer second. Then you can avoid storing 'NaN' and it should work fine.

But it's clear we should do something to fix this on our end, because I can easily see other people inadvertently running into this behavior. Options would include:

We'll have to think about it and figure out the best way forward. Thanks for trying things out and discovering this problem!

ryanvolz commented 5 years ago

The fix for the digital_rf_source py_io_signature bug has been released with Digital RF 2.6.2, so I'm going to close this. I'll open a new bug for discussion about the Digital RF Source block and NaNs.