search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
64 stars 29 forks source link

Python 3 ctypes compatibility #146

Closed mkolopanis closed 3 years ago

mkolopanis commented 3 years ago

I have been testing python 3 compatibility of an implementation of the EPIC code which uses bifrost to perform direct imaging from the EPIC Astronomy group.

Here the current software versions installed for ctypes, bifrost, and lsl (a full environment snapshot is listed below). I have run bifrost off of the latest master a53b35bb7d6507f9fb0564800f504040a2248bfa

(epic_py3) mkolopanis@intrepid [10:41:59] [~/src/LWA_EPIC/LWA] [script_consolidation *]
-> % python --version                                        
Python 3.6.12 :: Anaconda, Inc.
(epic_py3) mkolopanis@intrepid [10:42:08] [~/src/LWA_EPIC/LWA] [script_consolidation *]
-> % pip freeze | grep "ctypesgen\|bifrost\|lsl"
bifrost==0.9.0
ctypesgen @ git+https://github.com/olsonse/ctypesgen.git@9bd2d249aa4011c6383a10890ec6f203d7b7990f
lsl==2.0.2

Running the EPIC code results in a ctypes error during a ring resize call in bifrost. I am definitely open to the idea my environment was configured incorrectly but this seemed like the best place to get help.

(epic_py3) mkolopanis@intrepid [10:42:13] [~/src/LWA_EPIC/LWA] [script_consolidation *]
-> % python LWA_bifrost.py --offline --tbnfile=/data5/LWA_SV_data/data_raw/TBN/Jupiter/058161_000086727 --imagesize 64 --imageres 1.79057 --nts 512 --channels 4 --accumulate 50 --ints_per_file 40 --out_dir py3_test/no_dft/

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/mkolopanis/src/anaconda/envs/epic_py3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/mkolopanis/src/anaconda/envs/epic_py3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "LWA_bifrost.py", line 286, in main
    self.oring.resize(ogulp_size, buffer_factor=10)
  File "/home/mkolopanis/src/anaconda/envs/epic_py3/lib/python3.6/site-packages/bifrost-0.9.0-py3.6.egg/bifrost/ring.py", line 75, in resize
    nringlet) )
ctypes.ArgumentError: argument 2: <class 'TypeError'>: wrong type

^C2020-10-06 17:42:44 [WARNING ] Received signal 2 SIGINT
[1]    19264 killed     python LWA_bifrost.py --offline  --imagesize 64 --imageres 1.79057 --nts 512

full environment state

(epic_py3) mkolopanis@intrepid [10:57:45] [~/src/bifrost/python] [master *]
-> % lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic
(epic_py3) mkolopanis@intrepid [11:03:19] [~/src/bifrost/python] [master *]
-> % conda info   

     active environment : epic_py3
    active env location : /home/mkolopanis/src/anaconda/envs/epic_py3
            shell level : 2
       user config file : /home/mkolopanis/.condarc
 populated config files : /home/mkolopanis/.condarc
          conda version : 4.8.5
    conda-build version : 3.18.9
         python version : 3.6.12.final.0
       virtual packages : __cuda=10.2
                          __glibc=2.27
       base environment : /home/mkolopanis/src/anaconda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/mkolopanis/src/anaconda/pkgs
                          /home/mkolopanis/.conda/pkgs
       envs directories : /home/mkolopanis/src/anaconda/envs
                          /home/mkolopanis/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.5 requests/2.24.0 CPython/3.6.12 Linux/4.15.0-74-generic ubuntu/18.04.3 glibc/2.27
                UID:GID : 1046:1002
             netrc file : None
           offline mode : False
(epic_py3) mkolopanis@intrepid [10:57:05] [~/src/bifrost/python] [master *]
-> % conda list   
# packages in environment at /home/mkolopanis/src/anaconda/envs/epic_py3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
aipy                      3.0.1            py36hb3f55d8_0    conda-forge
astropy                   4.0.1.post1      py36h7b6447c_1  
atlas                     3.8.4                         0    free
attrs                     20.2.0                     py_0  
bifrost                   0.9.0                    pypi_0    pypi
blas                      1.0                         mkl  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2020.7.22                     0  
certifi                   2020.6.20                py36_0  
cfitsio                   3.470                hf0d0db6_6  
contextlib2               0.6.0.post1              pypi_0    pypi
ctypesgen                 0.1.1.post4              pypi_0    pypi
cycler                    0.10.0                   py36_0  
dbus                      1.13.16              hb2f20db_0  
ephem                     3.7.7.0                  pypi_0    pypi
expat                     2.2.9                he6710b0_2  
fftw                      3.3.8                h7b6447c_3  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.10.2               h5ab3b9f_0  
glib                      2.65.0               h3eb4bd4_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb31296c_0  
healpy                    1.14.0           py36h4b0bee4_0    conda-forge
icu                       58.2                 he6710b0_3  
importlib-metadata        1.7.0                    py36_0  
importlib-resources       3.0.0                    pypi_0    pypi
importlib_metadata        1.7.0                         0  
iniconfig                 1.0.1                      py_0  
intel-openmp              2020.2                      254  
jpeg                      9b                   h024ee3a_2  
kiwisolver                1.2.0            py36hfd86e86_0  
krb5                      1.18.2               h173b8e3_0  
lcms2                     2.11                 h396b838_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libcurl                   7.71.1               h20c2e04_1  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
libpng                    1.6.37               hbc83047_0  
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_1  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.14                 h7b6447c_0  
libxml2                   2.9.10               he19cac6_1  
lsl                       2.0.2                    pypi_0    pypi
lz4-c                     1.9.2                he6710b0_1  
matplotlib                3.3.1                         0  
matplotlib-base           3.3.1            py36h817c723_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py36he904b0f_0  
mkl_fft                   1.2.0            py36h23d657b_0  
mkl_random                1.1.1            py36h0573a6f_0  
more-itertools            8.5.0                      py_0  
ncurses                   6.2                  he6710b0_1  
numpy                     1.19.1           py36hbc911f0_0  
numpy-base                1.19.1           py36hfa32c7d_0  
olefile                   0.46                     py36_0  
openssl                   1.1.1h               h7b6447c_0  
packaging                 20.4                       py_0  
pcre                      8.44                 he6710b0_0  
pillow                    7.2.0            py36hb39fc2d_0  
pint                      0.16.1                   pypi_0    pypi
pip                       20.2.3                   py36_0  
pluggy                    0.13.1                   py36_0  
py                        1.9.0                      py_0  
pyephem                   3.7.7.0                  pypi_0    pypi
pyparsing                 2.4.7                      py_0  
pyqt                      5.9.2            py36h05f1152_2  
pytest                    6.1.0                    py36_0  
pytest-runner             5.2                        py_0  
python                    3.6.12               hcff3b4d_2  
python-dateutil           2.8.1                      py_0  
python-graphviz           0.14.1                   pypi_0    pypi
python_abi                3.6                     1_cp36m    conda-forge
pytz                      2020.1                     py_0  
qt                        5.9.7                h5867ecd_1  
readline                  8.0                  h7b6447c_0  
scipy                     1.5.2            py36h0b6359f_0  
setuptools                50.3.0           py36hb0f4dca_1  
simplejson                3.17.2           py36h7b6447c_0  
sip                       4.19.8           py36hf484d3e_0  
six                       1.15.0                     py_0  
sqlite                    3.33.0               h62c20be_0  
tk                        8.6.10               hbc83047_0  
toml                      0.10.1                     py_0  
tornado                   6.0.4            py36h7b6447c_1  
wheel                     0.35.1                     py_0  
xz                        5.2.5                h7b6447c_0  
zipp                      3.1.0                      py_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0  
(epic_py3) mkolopanis@intrepid [10:57:12] [~/src/bifrost/python] [master *]
-> % pip freeze
aipy==3.0.1
astropy==4.0.1.post1
attrs @ file:///tmp/build/80754af9/attrs_1600298409949/work
bifrost==0.9.0
certifi==2020.6.20
contextlib2==0.6.0.post1
ctypesgen @ git+https://github.com/olsonse/ctypesgen.git@9bd2d249aa4011c6383a10890ec6f203d7b7990f
cycler==0.10.0
ephem==3.7.7.0
graphviz==0.14.1
healpy @ file:///home/conda/feedstock_root/build_artifacts/healpy_1595518317472/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1593446433964/work
importlib-resources==3.0.0
iniconfig @ file:///tmp/build/80754af9/iniconfig_1596827328212/work
kiwisolver==1.2.0
lsl==2.0.2
matplotlib @ file:///tmp/build/80754af9/matplotlib-base_1597876339545/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
more-itertools @ file:///tmp/build/80754af9/more-itertools_1598884071673/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1596233737064/work
olefile==0.46
packaging==20.4
Pillow @ file:///tmp/build/80754af9/pillow_1594307298074/work
Pint==0.16.1
pluggy==0.13.1
py @ file:///tmp/build/80754af9/py_1593446248552/work
pyephem==3.7.7.0
pyparsing==2.4.7
pytest==0.0.0
pytest-runner==5.2
python-dateutil==2.8.1
pytz==2020.1
scipy @ file:///tmp/build/80754af9/scipy_1597686625380/work
simplejson @ file:///tmp/build/80754af9/simplejson_1598625952897/work
six==1.15.0
toml @ file:///tmp/build/80754af9/toml_1592853716807/work
tornado==6.0.4
zipp==3.1.0
jaycedowell commented 3 years ago

Could this be a true division vs. floor division problem in the EPIC code?

mkolopanis commented 3 years ago

oh an interesting idea. let me check.

mkolopanis commented 3 years ago

Looks like it was indeed a true division vs int division issue. Thanks for the insight.