Differences in Python 2 vs. Python 3 str objects cause many errors

[ejmichaud]

I am running Python 3.6.4 and have blimpy 1.1.9 installed. When I try to create a Waterfall object, I get the following error:

from blimpy import Waterfall
obs = Waterfall('voyager_f1032192_t300_v2.fil')

/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    obs = Waterfall('voyager_f1032192_t300_v2.fil')
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/waterfall.py", line 123, in __init__
    self.container = fw.open_file(filename, f_start=f_start, f_stop=f_stop,t_start=t_start, t_stop=t_stop,load_data=load_data,max_load=max_load)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/file_wrapper.py", line 861, in open_file
    return FIL_reader(filename,f_start=f_start, f_stop=f_stop,t_start=t_start, t_stop=t_stop,load_data=load_data,max_load=max_load)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/file_wrapper.py", line 451, in __init__
    self.header = self._read_header()
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/file_wrapper.py", line 637, in _read_header
    keyword, value, idx = self._read_next_header_keyword(fh)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/file_wrapper.py", line 601, in _read_next_header_keyword
    dtype = self._header_keyword_types[keyword]
KeyError: b'HEADER_START'

This seems to be a problem with how strings are treated in Python 2 vs. in Python 3. In Python 3, the "keyword" variable is assigned to an object of type "bytes", so when it is compared (on line 598) with the "str" object 'HEADER_START', it doesn't recognize them as being the same when it should. This is just one area where this difference causes problems. http://python3porting.com/preparing.html#richcomparisons

[telegraphic]

Thanks @ejmichaud -- we do need some nudging to move toward Py3!

Be aware that "there be dragons" with Py3, due to things like this, as we've predominantly used Py2. So if you're not adventurous / willing to debug / rip out some hair, I'd suggest sticking to Py2.

I think I fixed this in our sigproc.py file, which is used by Filterbank but not currently by Waterfall. This should thus be fixed in https://github.com/UCBerkeleySETI/blimpy/pull/36, which consolidated duplicated code.

Could I please get you to check one of two things (or both if you're keen): 1) does this work in Py3 with your current setup:

from blimpy import Filterbank
obs = Filterbank('voyager_f1032192_t300_v2.fil')

2) Does Waterfall work if you install blimpy from my dev repo: https://github.com/telegraphic/blimpy

Cheers, Danny

[ejmichaud]

Hey Danny,

Thanks for the swift response. Here are the results of each test.

1)

from blimpy import Filterbank
obs = Filterbank('voyager_f1032192_t300_v2.fil')

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    obs = Filterbank('voyager_f1032192_t300_v2.fil')
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/filterbank.py", line 122, in __init__
    self.read_filterbank(filename, f_start, f_stop, t_start, t_stop, load_data)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.1.9-py3.6.egg/blimpy/filterbank.py", line 273, in read_filterbank
    f.seek(ii_start * n_bits * n_ifs * n_chans / 8, 1)
TypeError: 'float' object cannot be interpreted as an integer

2) (installed by cloning your repo and running setup.py)

from blimpy import Waterfall
obs = Waterfall('voyager_f1032192_t300_v2.fil')

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    obs = Waterfall('voyager_f1032192_t300_v2.fil')
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.2.0-py3.6.egg/blimpy/waterfall.py", line 123, in __init__
    self.container = fw.open_file(filename, f_start=f_start, f_stop=f_stop,t_start=t_start, t_stop=t_stop,load_data=load_data,max_load=max_load)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.2.0-py3.6.egg/blimpy/file_wrapper.py", line 736, in open_file
    return FilReader(filename, f_start=f_start, f_stop=f_stop, t_start=t_start, t_stop=t_stop, load_data=load_data, max_load=max_load)
  File "/usr/local/Homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blimpy-1.2.0-py3.6.egg/blimpy/file_wrapper.py", line 457, in __init__
    self.n_channels_in_file  = self.header['nchans']
KeyError: 'nchans'

Best, Eric

[telegraphic]

Hey, yet more Py3 bugs: b'hello' != 'hello and 1 / 2 ==0 in Py2 but int(1) / int(2) == 0.5 in Py3, and array indexes need to be integers. Working on it.

[telegraphic]

Alrighty, I've made a bunch of changes on my branch, and the (still pretty basic) unit tests are passing on PY3. Could you please give it a whirl and see if it works?

[ejmichaud]

@telegraphic yes things are working great now! I am able to create a Waterfall object with no problems in both Python 2 and Python 3. Also, #32 seems to have been solved as well.

[telegraphic]

Phew! Closing this issue now, we can reopen a new one if (more like "when") bugs come up.