Py3ize

[bennahugo]

Alright @o-smirnov @twillis449 @landmanbester this is a first pass over cattery. I've run through various combinations of simulation switches and tested calico with some deep2 data to ensure we still have python 2 compatibility. However the beams modules of cattery should be tested with ddfacet. The pull request test suite should hopefully catch any glaring issues.

[bennahugo]

Alright basic checks have passed. I will test this further when timba itself is working on python 3

[bennahugo]

@twillis449 alright I've managed to change the backend to support unicode after some amount of head scratching and I've successfully calibrated. Unfortunately there is a bug in PyQt5 itself where it is missing some Qt5 API calls. This in turn makes Qwt fall over. Is there a necessity of using qt5 instead of 4?

[bennahugo]

Alright this seems to work in python 3 (kludge for the above mentioned issue is implemented in https://github.com/bennahugo/PythonQwt/blob/master/qwt/text.py:

I'll soon start PRs for all other repos

[bennahugo]

@gijzelaerr one step closer to a python 3 version. I still need to test this with a direction dependent calibration on 3C147 with the VLA beams to double check that I don't have non-obvious breakage in cattery.

[twillis449]

@bennahugo Plots look excellent :) I was under the impression that unicode was the default method for text implementation in python3 so I'm a bit puzzled by your struggles, but anyway ... :) As for the 'necessity' of going to PyQt5 vs PyQt4, there probably really wasn't one, but I just thought that if we had to make the great leap forward to python3 we might as well get 'caught up' with Qt releases as well. I believe that the Qt Company is hard at work on Qt6, so I suspect that support for Qt4 will stop in the not-too-distant future. If there are missing Qt APIs in PyQt5 that you think should be added, you should contact the PyQt mailing list. The developer is usually very responsive to this sort of issue. The email address for the list is pyqt@riverbankcomputing.com

[bennahugo]

Alright I'm happy that this works. I've done reasonably extensive testing in python 3. The next step is to merge all of the outstanding PRs.

[bennahugo]

• https://github.com/ska-sa/pyxis/pull/92
• https://github.com/ska-sa/kittens/pull/11
• https://github.com/ska-sa/owlcat/pull/55
• https://github.com/ska-sa/meqtrees-timba/pull/46

[bennahugo]

@sjperkins can you run your test suite against the various PRs listed up top? Note meqtrees does not work in a virtual environment because of the dynamic module loading magic.

[sjperkins]

@sjperkins can you run your test suite against the various PRs listed up top? Note meqtrees does not work in a virtual environment because of the dynamic module loading magic.

Is it possible to use a virtual environment with --system-site-packages? That's what I've always done in the past with the python2 versions.

[bennahugo]

No the module loading does not take into account the virtual environment paths

[bennahugo]

I've tried that with no success. I forgot about purr: https://github.com/ska-sa/purr/pull/34

[sjperkins]

Segfaults:

s$ $(which meqtree-pipeliner.py) -c meqtrees/tdlconf.profiles "[codex-compare-linear]" ms_sel.msname=/home/sperkins/data/WSRT.MS tiggerlsm.filename=../sky-model.txt ms_sel.output_column=CORRECTED_DATA me.e_enable=1 me.l_enable=1 pybeams_fits.filename_pattern="/tmp/beams-jn4bqkib/beams_\$(corr)_\$(reim).fits" pybeams_fits.l_axis=X pybeams_fits.m_axis=Y meqtrees/turbo-sim.py =simulate
Segmentation fault (core dumped)

$ gdb python core
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...Reading symbols from /usr/lib/debug/.build-id/1a/a107f8fdf4e3756f45758cc5a7b2155fe66eea.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 17840]
[New LWP 17841]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python /home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-r'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb70bb3bcfe in ?? () from /usr/lib/python2.7/dist-packages/PyQt4/QtCore.x86_64-linux-gnu.so
[Current thread is 1 (Thread 0x7fb732b48740 (LWP 17840))]
(gdb) bt
#0  0x00007fb70bb3bcfe in  () at /usr/lib/python2.7/dist-packages/PyQt4/QtCore.x86_64-linux-gnu.so
#1  0x00007fb73296a733 in call_init (env=0x1d7a310, argv=0x7ffe7f404f98, argc=15, l=<optimised out>)
    at dl-init.c:72
#2  0x00007fb73296a733 in _dl_init (main_map=main_map@entry=0x28b2050, argc=15, argv=0x7ffe7f404f98, env=0x1d7a310) at dl-init.c:119
#3  0x00007fb73296f1ff in dl_open_worker (a=a@entry=0x7ffe7f4036d0) at dl-open.c:522
#4  0x00007fb7324b12df in __GI__dl_catch_exception (exception=exception@entry=0x7ffe7f4036b0, operate=operate@entry=0x7fb73296edc0 <dl_open_worker>, args=args@entry=0x7ffe7f4036d0) at dl-error-skeleton.c:196
#5  0x00007fb73296e7ca in _dl_open (file=0x28b2e90 "/usr/lib/python2.7/dist-packages/PyQt4/QtCore.x86_64-linux-gnu.so", mode=-2147483390, caller_dlopen=0x51a006 <_PyImport_GetDynLoadFunc+214>, nsid=<optimised out>, argc=15, argv=<optimised out>, env=0x1d7a310) at dl-open.c:605
#6  0x00007fb732146f96 in dlopen_doit (a=a@entry=0x7ffe7f403900) at dlopen.c:66
#7  0x00007fb7324b12df in __GI__dl_catch_exception (exception=exception@entry=0x7ffe7f4038a0, operate=operate@entry=0x7fb732146f40 <dlopen_doit>, args=args@entry=0x7ffe7f403900) at dl-error-skeleton.c:196
#8  0x00007fb7324b136f in __GI__dl_catch_error (objname=objname@entry=0x1d54f00, errstring=errstring@entry=0x1d54f08, mallocedp=mallocedp@entry=0x1d54ef8, operate=operate@entry=0x7fb732146f40 <dlopen_doit>, args=args@entry=0x7ffe7f403900) at dl-error-skeleton.c:215
#9  0x00007fb732147735 in _dlerror_run (operate=operate@entry=0x7fb732146f40 <dlopen_doit>, args=args@entry=0x7ffe7f403900) at dlerror.c:162
#10 0x00007fb732147051 in __dlopen (file=<optimised out>, mode=<optimised out>) at dlopen.c:87
#11 0x000000000051a006 in _PyImport_GetDynLoadFunc (fqname=0x28b1040 "PyQt4.QtCore", shortname=0x28b1046 "QtCore", pathname=0x28b2e90 "/usr/lib/python2.7/dist-packages/PyQt4/QtCore.x86_64-linux-gnu.so", fp=0x20e4130) at ../Python/dynload_shlib.c:140
#12 0x0000000000519bf4 in _PyImport_LoadDynamicModule (name=0x28b1040 "PyQt4.QtCore", pathname=0x28b2e90 "/usr/lib/python2.7/dist-packages/PyQt4/QtCore.x86_64-linux-gnu.so", fp=0x20e4130)
    at ../Python/importdl.c:42
#13 0x00000000004a59d1 in import_submodule.lto_priv (mod=<module at remote 0x7fb710b518d8>, subname=0x28b1046 "QtCore", fullname=0x28b1040 "PyQt4.QtCore") at ../Python/import.c:2747
#14 0x00000000004a51dd in load_next (p_buflen=<synthetic pointer>, buf=<optimised out>, p_name=<synthetic pointer>, altmod=<module at remote 0x7fb710b518d8>, mod=<module at remote 0x7fb710b518d8>)
    at ../Python/import.c:2561
#15 0x00000000004a51dd in import_module_level.isra.3 (level=<optimised out>, fromlist=['__doc__'], globals=<optimised out>, name=<optimised out>) at ../Python/import.c:2278
#16 0x00000000004a51dd in PyImport_ImportModuleLevel (name=<optimised out>, globals=<optimised out>, locals=<optimised out>, fromlist=['__doc__'], level=<optimised out>) at ../Python/import.c:2334
#17 0x00000000004a6a44 in builtin___import__ (self=<optimised out>, args=<optimised out>, kwds=<optimised out>) at ../Python/bltinmodule.c:49
#18 0x00000000004a66b4 in PyObject_Call (kw=0x0, arg=('PyQt4.QtCore', {'make_hiid': <function at remote 0x7fb7241811b8>, 'arr_dcomplex': <type at remote 0x7fb731691800>, 'dmi_typename': <function at remote 0x7fb7241a4578>, 'arr_int32': <type at remote 0x7fb731693080>, 'refcount_report': <function at remote 0x7fb7241a46e0>, 'extract_stack': <function at remote 0x7fb724202320>, 'type_maker': <function at remote 0x7fb7242022a8>, 'Timba': <module at remote 0x7fb732a37750>, 'message': <type at remote 0x24c11e0>, 'array': <built-in function array>, 'weakref': <module at remote 0x7fb7306e8750>, 'copy': <module at remote 0x7fb72838cef8>, 'arr_double': <type at remote 0x7fb731692040>, 'is_array': <function at remote 0x7fb7241a4488>, 'six': <module at remote 0x7fb724186c90>, '__package__': 'Timba.Meq', 're': <module at remote 0x7fb732a62a28>, 'nonportable_extract_stack': <function at remote 0x7fb724202398>, 'WeakInstanceMethod': <type at remote 0x232db30>, 'is_scalar': <function at remote 0x7fb7241a4500>, 'xcurry': <function at remote 0x7fb724202500>, 'np': <module at re...(truncated), func=<built-in function __import__>)
    at ../Objects/abstract.c:2544
#19 0x00000000004a66b4 in call_function_tail.lto_priv.1801 (args=('PyQt4.QtCore', {'make_hiid': <function at remote 0x7fb7241811b8>, 'arr_dcomplex': <type at remote 0x7fb731691800>, 'dmi_typename': <function at remote 0x7fb7241a4578>, 'arr_int32': <type at remote 0x7fb731693080>, 'refcount_report': <function at remote 0x7fb7241a46e0>, 'extract_stack': <function at remote 0x7fb724202320>, 'type_maker': <function at remote 0x7fb7242022a8>, 'Timba': <module at remote 0x7fb732a37750>, 'message': <type at remote 0x24c11e0>, ---Type <return> to continue, or q <return> to quit---
'array': <built-in function array>, 'weakref': <module at remote 0x7fb7306e8750>, 'copy': <module at remote 0x7fb72838cef8>, 'arr_double': <type at remote 0x7fb731692040>, 'is_array': <function at remote 0x7fb7241a4488>, 'six': <module at remote 0x7fb724186c90>, '__package__': 'Timba.Meq', 're': <module at remote 0x7fb732a62a28>, 'nonportable_extract_stack': <function at remote 0x7fb724202398>, 'WeakInstanceMethod': <type at remote 0x232db30>, 'is_scalar': <function at remote 0x7fb7241a4500>, 'xcurry': <function at remote 0x7fb724202500>, 'np': <module at re...(truncated), callable=<optimised out>)
    at ../Objects/abstract.c:2576
#20 0x00000000004a66b4 in PyObject_CallFunction (callable=<built-in function __import__>, format=0x5c0e4f "OOOOi") at ../Objects/abstract.c:2600
#21 0x00000000004a48e6 in PyImport_Import (module_name='PyQt4.QtCore') at ../Python/import.c:2933
#22 0x00000000004a482a in PyImport_ImportModule (name=<optimised out>) at ../Python/import.c:2173
#23 0x00007fb70bf30679 in  () at /usr/lib/python2.7/dist-packages/PyQt4/Qt.x86_64-linux-gnu.so
#24 0x00007fb70bf306fc in initQt () at /usr/lib/python2.7/dist-packages/PyQt4/Qt.x86_64-linux-gnu.so
#25 0x0000000000519c21 in _PyImport_LoadDynamicModule (name=0x248d930 "PyQt4.Qt", pathname=0x248e940 "/usr/lib/python2.7/dist-packages/PyQt4/Qt.x86_64-linux-gnu.so", fp=0x241e480) at ../Python/importdl.c:53
#26 0x00000000004a59d1 in import_submodule.lto_priv (mod=<module at remote 0x7fb710b518d8>, subname=0x248d936 "Qt", fullname=0x248d930 "PyQt4.Qt") at ../Python/import.c:2747
#27 0x00000000004a51dd in load_next (p_buflen=<synthetic pointer>, buf=<optimised out>, p_name=<synthetic pointer>, altmod=<module at remote 0x7fb710b518d8>, mod=<module at remote 0x7fb710b518d8>)
    at ../Python/import.c:2561
#28 0x00000000004a51dd in import_module_level.isra.3 (level=<optimised out>, fromlist=('QObject', 'SIGNAL'), globals=<optimised out>, name=<optimised out>) at ../Python/import.c:2278
#29 0x00000000004a51dd in PyImport_ImportModuleLevel (name=<optimised out>, globals=<optimised out>, locals=<optimised out>, fromlist=('QObject', 'SIGNAL'), level=<optimised out>) at ../Python/import.c:2334
#30 0x00000000004a6a44 in builtin___import__ (self=<optimised out>, args=<optimised out>, kwds=<optimised out>) at ../Python/bltinmodule.c:49
#31 0x00000000004a67ee in PyObject_Call (func=<built-in function __import__>, arg=<optimised out>, kw=<optimised out>) at ../Objects/abstract.c:2544
#32 0x00000000004c6d50 in PyEval_CallObjectWithKeywords (func=<built-in function __import__>, arg=('PyQt4.Qt', {'make_hiid': <function at remote 0x7fb7241811b8>, 'arr_dcomplex': <type at remote 0x7fb731691800>, 'dmi_typename': <function at remote 0x7fb7241a4578>, 'arr_int32': <type at remote 0x7fb731693080>, 'refcount_report': <function at remote 0x7fb7241a46e0>, 'extract_stack': <function at remote 0x7fb724202320>, 'type_maker': <function at remote 0x7fb7242022a8>, 'Timba': <module at remote 0x7fb732a37750>, 'message': <type at remote 0x24c11e0>, 'array': <built-in function array>, 'weakref': <module at remote 0x7fb7306e8750>, 'copy': <module at remote 0x7fb72838cef8>, 'arr_double': <type at remote 0x7fb731692040>, 'is_array': <function at remote 0x7fb7241a4488>, 'six': <module at remote 0x7fb724186c90>, '__package__': 'Timba.Meq', 're': <module at remote 0x7fb732a62a28>, 'nonportable_extract_stack': <function at remote 0x7fb724202398>, 'WeakInstanceMethod': <type at remote 0x232db30>, 'is_scalar': <function at remote 0x7fb7241a4500>, 'xcurry': <function at remote 0x7fb724202500>, 'np': <module at remote...(truncated), kw=<optimised out>) at ../Python/ceval.c:4245
#33 0x00000000004bfe00 in PyEval_EvalFrameEx (f=<optimised out>, throwflag=<optimised out>)
    at ../Python/ceval.c:2647
#34 0x00000000004bad4a in PyEval_EvalCodeEx (co=0x7fb72416d6b0, globals=<optimised out>, locals=<optimised out>, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at ../Python/ceval.c:3608
#35 0x00000000004ba969 in PyEval_EvalCode (co=<optimised out>, globals=<optimised out>, locals=<optimised out>) at ../Python/ceval.c:669
#36 0x00000000004ba89f in PyImport_ExecCodeModuleEx (name=0x2481450 "Timba.Meq.meqds", co=<code at remote 0x7fb72416d6b0>, pathname=0x248ac50 "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Meq/meqds.pyc") at ../Python/import.c:753
#37 0x00000000004b3b03 in load_source_module.lto_priv (name=0x2481450 "Timba.Meq.meqds", pathname=0x248ac50 "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Meq/meqds.pyc", fp=<optimised out>) at ../Python/import.c:1143
#38 0x00000000004a59d1 in import_submodule.lto_priv (mod=<module at remote 0x7fb7241c3c90>, subname=0x7fb---Type <return> to continue, or q <return> to quit---
724206a74 "meqds", fullname=0x2481450 "Timba.Meq.meqds") at ../Python/import.c:2747
#39 0x00000000004a8fa5 in ensure_fromlist.lto_priv (mod=<module at remote 0x7fb7241c3c90>, fromlist=('meqds',), buf=0x2481450 "Timba.Meq.meqds", buflen=9, recursive=0) at ../Python/import.c:2653
#40 0x00000000004a4d69 in import_module_level.isra.3 (level=<optimised out>, fromlist=('meqds',), globals=<optimised out>, name=<optimised out>) at ../Python/import.c:2315
#41 0x00000000004a4d69 in PyImport_ImportModuleLevel (name=<optimised out>, globals=<optimised out>, locals=<optimised out>, fromlist=('meqds',), level=<optimised out>) at ../Python/import.c:2334
#42 0x00000000004a6a44 in builtin___import__ (self=<optimised out>, args=<optimised out>, kwds=<optimised out>) at ../Python/bltinmodule.c:49
#43 0x00000000004a67ee in PyObject_Call (func=<built-in function __import__>, arg=<optimised out>, kw=<optimised out>) at ../Objects/abstract.c:2544
#44 0x00000000004c6d50 in PyEval_CallObjectWithKeywords (func=<built-in function __import__>, arg=('Timba.Meq', {'__builtins__': {'bytearray': <type at remote 0x8ecf40>, 'IndexError': <type at remote 0x8f5880>, 'all': <built-in function all>, 'help': <_Helper at remote 0x7fb732a0bb90>, 'vars': <built-in function vars>, 'SyntaxError': <type at remote 0x8f5d60>, 'unicode': <type at remote 0x8ec240>, 'UnicodeDecodeError': <type at remote 0x8f5200>, 'memoryview': <type at remote 0x8eb200>, 'isinstance': <built-in function isinstance>, 'copyright': <_Printer(_Printer__data='Copyright (c) 2001-2019 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for National Research Initiatives.\nAll Rights Reserved.\n\nCopyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.\nAll Rights Reserved.', _Printer__lines=None, _Printer__name='copyright', _Printer__dirs=(), _Printer__files=(...)) at remote 0x7fb732a0b9d0>, 'NameError': <type at remote 0x8f6240>, 'BytesWarning': <type at remote 0x8f39a0>, 'dict': <type at remote ...(truncated), kw=<optimised out>) at ../Python/ceval.c:4245
#45 0x00000000004bfe00 in PyEval_EvalFrameEx (f=<optimised out>, throwflag=<optimised out>)
    at ../Python/ceval.c:2647
#46 0x00000000004bad4a in PyEval_EvalCodeEx (co=0x7fb7241a6d30, globals=<optimised out>, locals=<optimised out>, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at ../Python/ceval.c:3608
#47 0x00000000004ba969 in PyEval_EvalCode (co=<optimised out>, globals=<optimised out>, locals=<optimised out>) at ../Python/ceval.c:669
#48 0x00000000004ba89f in PyImport_ExecCodeModuleEx (name=0x1d1b930 "Timba.Apps.meqserver", co=<code at remote 0x7fb7241a6d30>, pathname=0x2480130 "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Apps/meqserver.pyc") at ../Python/import.c:753
#49 0x00000000004b3b03 in load_source_module.lto_priv (name=0x1d1b930 "Timba.Apps.meqserver", pathname=0x2480130 "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Apps/meqserver.pyc", fp=<optimised out>) at ../Python/import.c:1143
#50 0x00000000004a59d1 in import_submodule.lto_priv (mod=<module at remote 0x7fb724201168>, subname=0x7fb732a39da4 "meqserver", fullname=0x1d1b930 "Timba.Apps.meqserver") at ../Python/import.c:2747
#51 0x00000000004a8fa5 in ensure_fromlist.lto_priv (mod=<module at remote 0x7fb724201168>, fromlist=('meqserver',), buf=0x1d1b930 "Timba.Apps.meqserver", buflen=10, recursive=0) at ../Python/import.c:2653
#52 0x00000000004a4d69 in import_module_level.isra.3 (level=<optimised out>, fromlist=('meqserver',), globals=<optimised out>, name=<optimised out>) at ../Python/import.c:2315
#53 0x00000000004a4d69 in PyImport_ImportModuleLevel (name=<optimised out>, globals=<optimised out>, locals=<optimised out>, fromlist=('meqserver',), level=<optimised out>) at ../Python/import.c:2334
#54 0x00000000004a6a44 in builtin___import__ (self=<optimised out>, args=<optimised out>, kwds=<optimised out>) at ../Python/bltinmodule.c:49
#55 0x00000000004a67ee in PyObject_Call (func=<built-in function __import__>, arg=<optimised out>, kw=<optimised out>) at ../Objects/abstract.c:2544
#56 0x00000000004c6d50 in PyEval_CallObjectWithKeywords (func=<built-in function __import__>, arg=('Timba.Apps', {'rem_args': ['[codex-compare-linear]', 'ms_sel.msname=/home/sperkins/data/WSRT.MS', 'tiggerlsm.filename=../sky-model.txt', 'ms_sel.output_column=CORRECTED_DATA', 'me.e_enable=1', 'me.l_enable=1', 'pybeams_fits.filename_pattern=/tmp/beams-jn4bqkib/beams_$(corr)_$(reim).fits', 'pybeams_fits.l_axis=X', 'pybeams_fits.m_axis=Y', 'meqtrees/turbo-sim.py', '=simulate'], 'parser': <OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--trace': <Option(_long_opts=['--trace'], help='(for deb---Type <return> to continue, or q <return> to quit---
ugging Python code) enables line tracing of Python statements', callback_args=None, callback=None, default=('NO', 'DEFAULT'), nargs=None, choices=None, dest='trace', container=<...>, _short_opts=['-t'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7fb7241f9560>, '--save-config': <Option(_long_opts=['--save-config'], help='save final configuration (after all command-line arguments have been applied) to config file and section', callback_args=...(truncated), kw=<optimised out>) at ../Python/ceval.c:4245
#57 0x00000000004bfe00 in PyEval_EvalFrameEx (f=<optimised out>, throwflag=<optimised out>)
    at ../Python/ceval.c:2647
#58 0x00000000004bad4a in PyEval_EvalCodeEx (co=0x7fb732a1c1b0, globals=<optimised out>, locals=<optimised out>, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at ../Python/ceval.c:3608
#59 0x00000000004ba969 in PyEval_EvalCode (co=<optimised out>, globals=<optimised out>, locals=<optimised out>) at ../Python/ceval.c:669
#60 0x00000000004eb0af in run_mod.lto_priv (mod=0x1d189f8, filename=0x7ffe7f4054fe "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/meqtree-pipeliner.py", globals={'rem_args': ['[codex-compare-linear]', 'ms_sel.msname=/home/sperkins/data/WSRT.MS', 'tiggerlsm.filename=../sky-model.txt', 'ms_sel.output_column=CORRECTED_DATA', 'me.e_enable=1', 'me.l_enable=1', 'pybeams_fits.filename_pattern=/tmp/beams-jn4bqkib/beams_$(corr)_$(reim).fits', 'pybeams_fits.l_axis=X', 'pybeams_fits.m_axis=Y', 'meqtrees/turbo-sim.py', '=simulate'], 'parser': <OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--trace': <Option(_long_opts=['--trace'], help='(for debugging Python code) enables line tracing of Python statements', callback_args=None, callback=None, default=('NO', 'DEFAULT'), nargs=None, choices=None, dest='trace', container=<...>, _short_opts=['-t'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7fb7241f9560>, '--save-config': <Option(_long_opts=['--save-config'], help='save final configuration (after all command-line arguments have been applied) to config file and section', callback_args=None, callback=...(truncated), locals={'rem_args': ['[codex-compare-linear]', 'ms_sel.msname=/home/sperkins/data/WSRT.MS', 'tiggerlsm.filename=../sky-model.txt', 'ms_sel.output_column=CORRECTED_DATA', 'me.e_enable=1', 'me.l_enable=1', 'pybeams_fits.filename_pattern=/tmp/beams-jn4bqkib/beams_$(corr)_$(reim).fits', 'pybeams_fits.l_axis=X', 'pybeams_fits.m_axis=Y', 'meqtrees/turbo-sim.py', '=simulate'], 'parser': <OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--trace': <Option(_long_opts=['--trace'], help='(for debugging Python code) enables line tracing of Python statements', callback_args=None, callback=None, default=('NO', 'DEFAULT'), nargs=None, choices=None, dest='trace', container=<...>, _short_opts=['-t'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7fb7241f9560>, '--save-config': <Option(_long_opts=['--save-config'], help='save final configuration (after all command-line arguments have been applied) to config file and section', callback_args=None, callback=...(truncated), flags=0x7ffe7f404de0, arena=0x1c8baa0) at ../Python/pythonrun.c:1385
#61 0x00000000004e5b92 in PyRun_FileExFlags (fp=0x1cf9940, filename=0x7ffe7f4054fe "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/meqtree-pipeliner.py", start=257, globals={'rem_args': ['[codex-compare-linear]', 'ms_sel.msname=/home/sperkins/data/WSRT.MS', 'tiggerlsm.filename=../sky-model.txt', 'ms_sel.output_column=CORRECTED_DATA', 'me.e_enable=1', 'me.l_enable=1', 'pybeams_fits.filename_pattern=/tmp/beams-jn4bqkib/beams_$(corr)_$(reim).fits', 'pybeams_fits.l_axis=X', 'pybeams_fits.m_axis=Y', 'meqtrees/turbo-sim.py', '=simulate'], 'parser': <OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--trace': <Option(_long_opts=['--trace'], help='(for debugging Python code) enables line tracing of Python statements', callback_args=None, callback=None, default=('NO', 'DEFAULT'), nargs=None, choices=None, dest='trace', container=<...>, _short_opts=['-t'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7fb7241f9560>, '--save-config': <Option(_long_opts=['--save-config'], help='save final configuration (after all command-line arguments have been applied) to config file and section', callback_args=None, callback=...(truncated), locals={'rem_args': ['[codex-compare-linear]', 'ms_sel.msname=/home/sperkins/data/WSRT.MS', 'tiggerlsm.filename=../sky-model.txt', 'ms_sel.output_column=CORRECTED_DATA', 'me.e_enable=1', 'me.l_enable=1', 'pybeams_fits.filename_pattern=/tmp/beams-jn4bqkib/beams_$(corr)_$(reim).fits', 'pybeams_fits.l_axis=X', 'pybeams_fits.m_axis=Y', 'meqtrees/turbo-sim.py', '=simulate'], 'parser': <OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--trace': <Option(_long_opts=['--trace'], help='(for debugging Python code) enables line tracing of Python statements', callback_args=None, callback=None, default=('NO', 'DE---Type <return> to continue, or q <return> to quit---
FAULT'), nargs=None, choices=None, dest='trace', container=<...>, _short_opts=['-t'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7fb7241f9560>, '--save-config': <Option(_long_opts=['--save-config'], help='save final configuration (after all command-line arguments have been applied) to config file and section', callback_args=None, callback=...(truncated), closeit=1, flags=0x7ffe7f404de0) at ../Python/pythonrun.c:1371
#62 0x00000000004e5666 in PyRun_SimpleFileExFlags (fp=0x1cf9940, filename=<optimised out>, closeit=1, flags=0x7ffe7f404de0) at ../Python/pythonrun.c:957
#63 0x0000000000494bc0 in Py_Main (argc=<optimised out>, argv=<optimised out>) at ../Modules/main.c:645
#64 0x00007fb73236bb97 in __libc_start_main (main=
    0x4945f0 <main>, argc=15, argv=0x7ffe7f404f98, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffe7f404f88) at ../csu/libc-start.c:310
#65 0x0000000000494519 in _start ()

[bennahugo]

Are you running this with python 3 because it is binding to qt4 in python 2. I think you have a mismatch in python versions and compilation going on here. Also you should probably run it as python3 $(which meqtree-pipeliner.py)

[sjperkins]

Please give me the command for building timba with python3 enabled?

[bennahugo]

cmake -DENABLE_PYTHON_3=ON ../../

[bennahugo]

you need to install python3-pyqt4 python3-pyqt5 and my branch of pyqwt: https://github.com/bennahugo/PythonQwt

[bennahugo]

and also python3-sip

[sjperkins]

Compilied with python3 enabled:

$ $(which meqtree-pipeliner.py) -c meqtrees/tdlconf.profiles "[codex-compare-linear]" ms_sel.msname=/home/sperkins/data/WSRT.MS tiggerlsm.filename=../sky-model.txt ms_sel.output_column=CORRECTED_DATA me.e_enable=1 me.l_enable=1 pybeams_fits.filename_pattern="/tmp/beams-jn4bqkib/beams_\$(corr)_\$(reim).fits" pybeams_fits.l_axis=X pybeams_fits.m_axis=Y meqtrees/turbo-sim.py =simulate
=========================================================================
======= Error importing octopython module:
Traceback (most recent call last):
  File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/dmi.py", line 619, in <module>
    import Timba.octopython
ImportError: dynamic module does not define init function (initoctopython)
======= Running Timba.dmi module stand-alone with limited functionality
======= (some things may fail)
=========================================================================
Traceback (most recent call last):
  File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/meqtree-pipeliner.py", line 71, in <module>
    from Timba.Apps import meqserver
  File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Apps/meqserver.py", line 32, in <module>
    from Timba.Meq import meqds
  File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Meq/meqds.py", line 33, in <module>
    from Timba import mequtils
ImportError: dynamic module does not define init function (initmequtils)

[bennahugo]

Should be

python3 $(which meqtree-pipeliner.py).

On Tue, Nov 12, 2019 at 10:34 AM Simon Perkins notifications@github.com wrote:

Compilied with python3 enabled:

$ $(which meqtree-pipeliner.py) -c meqtrees/tdlconf.profiles "[codex-compare-linear]" ms_sel.msname=/home/sperkins/data/WSRT.MS tiggerlsm.filename=../sky-model.txt ms_sel.output_column=CORRECTED_DATA me.e_enable=1 me.l_enable=1 pybeams_fits.filenamepattern="/tmp/beams-jn4bqkib/beams\$(corr)_\$(reim).fits" pybeams_fits.l_axis=X pybeams_fits.m_axis=Y meqtrees/turbo-sim.py =simulate

======= Error importing octopython module: Traceback (most recent call last): File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/dmi.py", line 619, in import Timba.octopython ImportError: dynamic module does not define init function (initoctopython) ======= Running Timba.dmi module stand-alone with limited functionality ======= (some things may fail)

Traceback (most recent call last): File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/meqtree-pipeliner.py", line 71, in from Timba.Apps import meqserver File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Apps/meqserver.py", line 32, in from Timba.Meq import meqds File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Meq/meqds.py", line 33, in from Timba import mequtils ImportError: dynamic module does not define init function (initmequtils)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meqtrees-cattery/pull/102?email_source=notifications&email_token=AB4RE6TM5J74FIJZESTQYP3QTJTBXA5CNFSM4JHMLUAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDZOMNQ#issuecomment-552789558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VMRZ7GLA73RPKT5I3QTJTBXANCNFSM4JHMLUAA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[sjperkins]

$ python3 $(which meqtree-pipeliner.py) -c meqtrees/tdlconf.profiles "[codex-compare-linear]" ms_sel.msname=/home/sperkins/data/WSRT.MS tiggerlsm.filename=../sky-model.txt ms_sel.output_column=CORRECTED_DATA me.e_enable=1 me.l_enable=1 pybeams_fits.filename_pattern="/tmp/beams-o31fjf7d/beams_\$(corr)_\$(reim).fits" pybeams_fits.l_axis=X pybeams_fits.m_axis=Y meqtrees/turbo-sim.py =simulate
  File "/home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/meqtree-pipeliner.py", line 172
    try:
       ^
TabError: inconsistent use of tabs and spaces in indentation

$ ls -l /home/sperkins/work/ska/code/meqtrees/meqtrees-timba/install/symlinked-release/bin/
total 4
lrwxrwxrwx 1 sperkins sperkins 54 Aug 17  2016 addbitflagcol -> ../../../build/release/AppAgent/AppUtils/addbitflagcol
lrwxrwxrwx 1 sperkins sperkins 54 Aug 17  2016 addtiledmscol -> ../../../build/release/AppAgent/AppUtils/addtiledmscol
lrwxrwxrwx 1 sperkins sperkins 30 Aug 17  2016 gprof-run -> ../../../TimBase/src/gprof-run
lrwxrwxrwx 1 sperkins sperkins 13 Aug 17  2016 meqbrowser -> meqbrowser.py
lrwxrwxrwx 1 sperkins sperkins 33 Aug 17  2016 meqbrowser.py -> ../../../PyApps/src/meqbrowser.py
lrwxrwxrwx 1 sperkins sperkins 42 Aug 17  2016 meqserver -> ../../../build/release/MeqServer/meqserver
-rwxrwxr-x 1 sperkins sperkins 97 Aug 17  2016 meqserver-mpi
lrwxrwxrwx 1 sperkins sperkins 40 Aug 17  2016 meqtree-pipeliner.py -> ../../../PyApps/src/meqtree-pipeliner.py
lrwxrwxrwx 1 sperkins sperkins 32 Aug 17  2016 purr.py -> ../../../PyApps/src/Purr/purr.py
lrwxrwxrwx 1 sperkins sperkins 29 Aug 17  2016 tdlrun.py -> ../../../PyApps/src/tdlrun.py
lrwxrwxrwx 1 sperkins sperkins 24 Aug 17  2016 trut -> ../../../PyApps/src/trut
lrwxrwxrwx 1 sperkins sperkins 27 Aug 17  2016 trutify -> ../../../PyApps/src/trutify
lrwxrwxrwx 1 sperkins sperkins 27 Aug 17  2016 trut.py -> ../../../PyApps/src/trut.py
lrwxrwxrwx 1 sperkins sperkins 55 Aug 17  2016 wsrt_j2convert -> ../../../build/release/AppAgent/AppUtils/wsrt_j2convert

[sjperkins]

Users will need to manually 2to3 their turbo-sims.py too.

[bennahugo]

That is if they have custom scripts (but why would you have custom scripts)? I've found the residual issues and will commit just now

[bennahugo]

@sjperkins please pull timba (no need to rebuild)

[bennahugo]

@o-smirnov need your eyes here. The meqtrees test does not catch this:

CASA <1>: import numpy as np

CASA <2>: tb.open("/home/bhugo/workspace/WSRT.MS/")
Out[2]: True

CASA <3>: c = tb.getcol("CORRECTED_DATA")

CASA <4>: np.sum(np.all(c == 0, axis=(0,1)))
Out[4]: 1008

CASA <5>: np.sum(np.all(c != 0, axis=(0,1)))
Out[5]: 6552

it looks like there is an error in the tile handling logic

[bennahugo]

This happens regardless of whether the beams are on or not

[bennahugo]

Nevermind. I'm being stupid. I forget none of these tools actually predict the auto powers

[bennahugo]

Comparing codex-africanus with the current Kern 5 release with beams:

CASA <12>: np.max(np.abs(m[:,:,a1!=a2] - c[:,:,a1!=a2]))
Out[12]: 9.3132257461547852e-10
CASA <14>: np.rad2deg(np.max(np.angle(m[:,:,a1!=a2] - c[:,:,a1!=a2])))
Out[14]: 180.0

The current python3 masters:

CASA <9>: np.rad2deg(np.max(np.angle(m[:,:,a1!=a2] - c[:,:,a1!=a2])))
Out[9]: 180.0

CASA <10>: np.max(np.abs(m[:,:,a1!=a2] - c[:,:,a1!=a2]))
Out[10]: 9.3132257461547852e-10

This hog is airborne!