Meqtrees: Timba.mequtils not defined

[kamva1]

=========================================================================
======= Error importing octopython module:
Traceback (most recent call last):
  File "/home/vanqa/Vanqa/KamvaMSC2021/REAL_MSC_WORK/BUILD/meqtrees-timba/install/symlinked-release/libexec/python/Timba/dmi.py", line 619, in <module>
    import Timba.octopython
ImportError: No module named octopython
======= Running Timba.dmi module stand-alone with limited functionality
======= (some things may fail)
=========================================================================
Traceback (most recent call last):
  File "/usr/bin/meqtree-pipeliner.py", line 71, in <module>
    from Timba.Apps import meqserver
  File "/home/vanqa/Vanqa/KamvaMSC2021/REAL_MSC_WORK/BUILD/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Apps/meqserver.py", line 32, in <module>
    from Timba.Meq import meqds
  File "/home/vanqa/Vanqa/KamvaMSC2021/REAL_MSC_WORK/BUILD/meqtrees-timba/install/symlinked-release/libexec/python/Timba/Meq/meqds.py", line 33, in <module>
    from Timba import mequtils
ImportError: cannot import name mequtils
Command exited with non-zero status 1
2.65user 3.31system 0:00.78elapsed 758%CPU (0avgtext+0avgdata 99368maxresident)k
0inputs+16outputs (0major+13499minor)pagefaults 0swaps
2022/06/07 18:04:06 ABORT: meqtree-pipeliner.py returned error code 1 

I get this when I run meqtrees on the virtual environment (python 3.6) on the server jake n ike.

[kamva1]

Using LSM module from Tigger (1.4.2) 1.4.2 at /usr/lib/python2.7/dist-packages/Tigger (in path)
Successful readonly open of default-locked table MeerKAT_Hires.MS_p0: 25 columns, 151200 rows
Successful readonly open of default-locked table MeerKAT_Hires.MS_p0: 25 columns, 151200 rows
Successful readonly open of default-locked table MeerKAT_Hires.MS_p0: 25 columns, 151200 rows
Successful readonly open of default-locked table MeerKAT_Hires.MS_p0: 25 columns, 151200 rows
('### ', 'TDL script successfully compiled. 1414 node definitions\n  (of which 3 are root nodes) sent to meqserver.')
### Running TDL job "_tdl_job_1_simulate_MS"
Meow.ReadVisHeader
('### Job result:', None)
### Stopping the meqserver
### All your batch are belong to us. Bye!
27.70user 17.58system 0:33.61elapsed 134%CPU (0avgtext+0avgdata 419112maxresident)k
3522704inputs+3402216outputs (284major+503865minor)pagefaults 0swaps
2022/06/07 19:58:25 PYXIS: meqtree-pipeliner.py succeeded
Visibility column shape: (151200, 360, 4)
Successful readonly open of default-locked table /home/vanqa/Vanqa/Vanqa/MeerKAT_Hires.MS_p0/SPECTRAL_WINDOW: 14 columns, 1 rows
2022/06/07 19:58:58 PYXIS: registered module 'im.lwimager'
Traceback (most recent call last):
  File "sim_time.py", line 138, in <module>
    sim_now()
  File "sim_time.py", line 124, in sim_now
    weight = "natural")#,basename_image='sim_img_%d.fits'%k, dirty_image = 'sim_img_%d.fits'%k);
  File "/usr/lib/python2.7/dist-packages/Pyxides/imager.py", line 92, in make_image
    call_imager = eval( 'im.%s.make_image'%(imager.lower()) )
  File "<string>", line 1, in <module>
NameError: name 'im' is not defined

Then this is the error I get when I run meqtrees straight from the server(like n jake) python 2.7

[bennahugo]

The system packages for 18.04 are old and are not compatible with Python 3 - at least not for 18.04

If you want Python 3 support you need to compile and install the latest releases. See https://github.com/ratt-ru/meqtrees/wiki/BuildFromSource

If you can't compile the packages successfully feel free to open a ticket

[bennahugo]

Also before I forget - the newer versions of meqtrees-timba will not compile against the Ubuntu 18.04 shipped packages of casacore and casarest. You will need to grab the latest releases for them to compile against.

There is a Ubuntu 20.04 shipment that is supported. See https://github.com/ratt-ru/meqtrees-timba/blob/master/.ci/py3.docker

[kamva1]

Yes this is the one I was installing, which does not have Timba.mequtils.

On Tue, Jun 7, 2022 at 8:57 PM Benjamin Hugo @.***> wrote:

The system packages for 18.04 are old and are not compatible with Python 3

• at least not for 18.04

If you want Python 3 support you need to compile and install the latest releases. See https://github.com/ratt-ru/meqtrees/wiki/BuildFromSource

If you can't compile the packages successfully feel free to open a ticket

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/meqtrees/issues/887#issuecomment-1149049193, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARAQXKCBJGQECUC3KSPTRPDVN6LQ3ANCNFSM5YDZHJFQ . You are receiving this because you authored the thread.Message ID: @.***>

[bennahugo]

Your backtraces show you are using system packages. If you are compiling from source on a system that have previous versions installed you will need to isolate the python3 virtual environment (ie. not using --system-site-packages)

[kamva1]

I created my python3 virtual environment without ''system-site-packages''. I just uninstalled it and re -installed but still the same problem.

[o-smirnov]

Which node are you on @kamva1. Please post the sequence of commands that you are running -- I will try to log in as you and recreate the problem.

[bennahugo]

Please double check that you are not running the system packages (ie. venv/bin must be first in the path and load library path should be set to venv/lib). You can look at the output of which and ldd to check that this is the case.

On Wed, Jun 8, 2022 at 10:23 AM Oleg Smirnov @.***> wrote:

Which node are you on @kamva1 https://github.com/kamva1. Please post the sequence of commands that you are running -- I will try to log in as you and recreate the problem.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/meqtrees/issues/887#issuecomment-1149613138, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6TICD6ECWUJELKUDILVOBJ73ANCNFSM5YDZHJFQ . You are receiving this because you modified the open/close state.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[kamva1]

@o-smirnov using Jake, I was following the instructions from these installation instructions. I tried this method of building but didn't work.

cd $BUILD/meqtrees-timba
mkdir build
Tools/Build/bootstrap_cmake debug
cd build/debug # from  the instruction it's supposed to be [cd build/release], but I changed it to debug because the release was not formed  
make -j7

then including $BUILD/meqtrees-timba/install/symlinked-release/libexec/python to your PYTHONPATH and $BUILD/meqtrees-timba/install/symlinked-release/bin to PATH after virtualenv.

Also, I tried the 2nd method of building, but also failed with same error.

 $ PYTHONVER=3.6 # or 3.8 if you initialized your venv with 3.8
 $ cd $BUILD/meqtrees-timba
 $ mkdir build
 $ cd build
 $ cmake  -DCMAKE_INSTALL_PREFIX=$BUILD/venvmt \
          -DCMAKE_BUILD_TYPE=Release \
          -DENABLE_PYTHON_3=ON \
          -DPYTHON_PACKAGES_DIR=/$BUILD/venvmt/lib/python${PYTHONVER}/site-packages
 $ make install -jN

While running make on both methods, we had to comment out these lines because they were causing make to fail

https://github.com/ratt-ru/meqtrees-timba/blob/600139db0859edb6ca5b9d0bf678488c1e57affc/AppAgent/AppUtils/src/wsrt_j2convert.cc#L60-#L68

Thirdly, I tried to follow what @bennahugo suggested

There is a Ubuntu 20.04 shipment that is supported. See https://github.com/ratt-ru/meqtrees-timba/blob/master/.ci/py3.docker

and the ldconfig gives me the following /sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: Permission denied.

[bennahugo]

Hi the reason why make fails is because you are likely trying to compile against the system casacore and casarest versions. As mentioned you have to pull and build the latest of these packages before building meqtrees.

See: https://github.com/ratt-ru/meqtrees-timba/blob/master/cmake/FindCasaCore.cmake#L27 and https://github.com/ratt-ru/meqtrees-timba/blob/master/cmake/FindCasarest.cmake#L12

to specify your custom compiled casacore and casarest packages

On Wed, Jun 8, 2022 at 10:55 AM kamva1 @.***> wrote:

@o-smirnov https://github.com/o-smirnov using Jake, I was following the instructions from these installation instructions https://github.com/ratt-ru/meqtrees/wiki/BuildFromSource. I tried this method of building but didn't work.

cd $BUILD/meqtrees-timba mkdir build Tools/Build/bootstrap_cmake debug cd build/debug # from the instruction it's supposed to be [cd build/release], but I changed it to debug because the release was not formed make -j7

then including $BUILD/meqtrees-timba/install/symlinked-release/libexec/python to your PYTHONPATH and $BUILD/meqtrees-timba/install/symlinked-release/bin to PATH after virtualenv.

Also, I tried the 2nd method of building, but also failed with same error.

$ PYTHONVER=3.6 # or 3.8 if you initialized your venv with 3.8 $ cd $BUILD/meqtrees-timba $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX=$BUILD/venvmt \ -DCMAKE_BUILD_TYPE=Release \ -DENABLE_PYTHON_3=ON \ -DPYTHON_PACKAGES_DIR=/$BUILD/venvmt/lib/python${PYTHONVER}/site-packages $ make install -jN

While running make on both methods, we had to comment out these lines because they were causing make to fail

https://github.com/ratt-ru/meqtrees-timba/blob/600139db0859edb6ca5b9d0bf678488c1e57affc/AppAgent/AppUtils/src/wsrt_j2convert.cc#L60-#L68

Thirdly, I tried to follow what @bennahugo https://github.com/bennahugo suggested

There is a Ubuntu 20.04 shipment that is supported. See https://github.com/ratt-ru/meqtrees-timba/blob/master/.ci/py3.docker

and the ldconfig gives me the following /sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: Permission denied.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/meqtrees/issues/887#issuecomment-1149645378, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6QSWMY2QA2L23MQV5DVOBNXLANCNFSM5YDZHJFQ . You are receiving this because you were mentioned.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[Mulan-94]

Hey @bennahugo it looked like it was failing because of a syntax error I guess. So I advised editing it out, See the error :

/home/meqtrees-timba/AppAgent/AppUtils/src/wsrt_j2convert.cc:65:7: error: #elif with no expression
  #elif
       ^
compilation terminated due to -Wfatal-errors.
AppAgent/AppUtils/CMakeFiles/wsrt_j2convert.dir/build.make:62: recipe for target 'AppAgent/AppUtils/CMakeFiles/wsrt_j2convert.dir/src/wsrt_j2convert.cc.o' failed
make[2]: *** [AppAgent/AppUtils/CMakeFiles/wsrt_j2convert.dir/src/wsrt_j2convert.cc.o] Error 1
CMakeFiles/Makefile2:802: recipe for target 'AppAgent/AppUtils/CMakeFiles/wsrt_j2convert.dir/all' failed

Meanwhile, we will try compiling casacore and casarest from source

[o-smirnov]

OK hang on, there are two separate problems here so the conversation is getting confusing. Lwt's try to straighten this out. @kamva1 please open a separate issue for this one:

2022/06/07 19:58:58 PYXIS: registered module 'im.lwimager'
Traceback (most recent call last):
  File "sim_time.py", line 138, in <module>
    sim_now()
  File "sim_time.py", line 124, in sim_now
    weight = "natural")#,basename_image='sim_img_%d.fits'%k, dirty_image = 'sim_img_%d.fits'%k);
  File "/usr/lib/python2.7/dist-packages/Pyxides/imager.py", line 92, in make_image
    call_imager = eval( 'im.%s.make_image'%(imager.lower()) )
  File "<string>", line 1, in <module>
NameError: name 'im' is not defined

...and give me the exact and complete sequence of commands which lead to this error, so I can log in as you and recreate the problem from scratch.

This NameError: name 'im' is not defined problem appears to have popped up after an apt upgrade of the packages, so I'd like to understand what went wrong.

[bennahugo]

Ok the ticket originally said that they wanted to use this within a python3.6 virtualenv, which would not be feasible with the old 2.7 packages. So the issue is perhaps confused.

However, 2.7 is no longer supported, so I suggest we move on check if this is an issue with the latest releases.

On Wed, Jun 8, 2022 at 11:37 AM Oleg Smirnov @.***> wrote:

OK hang on, there are two separate problems here so the conversation is getting confusing. Lwt's try to straighten this out. @kamva1 https://github.com/kamva1 please open a separate issue for this one:

2022/06/07 19:58:58 PYXIS: registered module 'im.lwimager' Traceback (most recent call last): File "sim_time.py", line 138, in sim_now() File "sim_time.py", line 124, in sim_now weight = "natural")#,basename_image='simimg%d.fits'%k, dirty_image = 'simimg%d.fits'%k); File "/usr/lib/python2.7/dist-packages/Pyxides/imager.py", line 92, in make_image call_imager = eval( 'im.%s.make_image'%(imager.lower()) ) File "", line 1, in NameError: name 'im' is not defined

...and give me the exact and complete sequence of commands which lead to this error, so I can log in as you and recreate the problem from scratch.

This NameError: name 'im' is not defined problem appears to have popped up after an apt upgrade of the packages, so I'd like to understand what went wrong.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/meqtrees/issues/887#issuecomment-1149691590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VOPA6YMLDVJKIVNZ3VOBSVTANCNFSM5YDZHJFQ . You are receiving this because you were mentioned.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[o-smirnov]

Yep, that's why I want @kamva1's command sequence. It seems the 2.7 KERN versions were pottering along ok, until I started updating the system.

[bennahugo]

Ok I can confirm there is a build issue not picked up with shipped releases of casacore 3.4 and ccrest 1.8 under Ubuntu 20.04. I will make a patch for that now. There is however also build issues when an old casacore is already present in the /usr/ directories (the compiler is spitting out warnings... I will need to patch that as well). The current version of timba will not work with older versions of casacore due to deprecations being enforced inside cascacore (and which I now support with the latest release).

[bennahugo]

Patch to be applied

rom 0d1b6d661976707694c3f61323349af98311ac23 Mon Sep 17 00:00:00 2001
From: Benna Hugo <bennahugo@aol.com>
Date: Wed, 8 Jun 2022 13:25:15 +0200
Subject: [PATCH] Add include paths from findcasacore/rest

This fixes a problem with outdated include paths when
compiling on a system with both custom and system casacore/rest packages
---
 AppAgent/AppUtils/CMakeLists.txt        | 2 +-
 AppAgent/AppUtils/src/wsrt_j2convert.cc | 2 +-
 CMakeLists.txt                          | 2 ++
 3 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/AppAgent/AppUtils/CMakeLists.txt b/AppAgent/AppUtils/CMakeLists.txt
index eb8ae826..85e7c1b9 100644
--- a/AppAgent/AppUtils/CMakeLists.txt
+++ b/AppAgent/AppUtils/CMakeLists.txt
@@ -20,7 +20,7 @@ set(appUtils_src
     )
 add_library(apputils ${appUtils_src})
 install(TARGETS apputils DESTINATION ${LIBRARY_INSTALL_DIR})
-target_link_libraries(apputils casa_ms)
+target_link_libraries(apputils ${CASACORE_LIBRARIES})
 MEQPACKAGE_ADD_LIBRARIES(apputils)

 add_executable(addbitflagcol src/addbitflagcol.cc)
diff --git a/AppAgent/AppUtils/src/wsrt_j2convert.cc b/AppAgent/AppUtils/src/wsrt_j2convert.cc
index 8fe280d3..002de793 100644
--- a/AppAgent/AppUtils/src/wsrt_j2convert.cc
+++ b/AppAgent/AppUtils/src/wsrt_j2convert.cc
@@ -62,7 +62,7 @@
 #else
        #if CASACORE_MINOR_VERSION >= 4
                #include <casacore/casa/IO/ArrayIO.h>
-       #elif
+       #else
                #include <casacore/casa/Arrays/ArrayIO.h>
        #endif
 #endif
diff --git a/CMakeLists.txt b/CMakeLists.txt
index c268596c..1074a45f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -110,6 +110,7 @@ if(CASACORE_FOUND)
     add_definitions(-DHAVE_AIPSPP)
     message("-- Casacore include directories found in ${CASACORE_INCLUDE_DIR}")
     message("-- Casacore libraries found in ${CASACORE_LIBRARIES}")
+    include_directories(${CASACORE_INCLUDE_DIR})
 endif(CASACORE_FOUND)

 find_package(Casarest REQUIRED COMPONENTS synthesis)
@@ -117,6 +118,7 @@ find_package(Casarest REQUIRED COMPONENTS synthesis)
 if(CASAREST_FOUND)
     message("-- Casarest include directories found in ${CASAREST_INCLUDE_DIR}")
     message("-- Casarest libraries found in ${CASAREST_LIBRARIES}")
+    include_directories(${CASAREST_INCLUDE_DIR})
 endif(CASAREST_FOUND)

 # -- compiler defaults
-- 
2.17.1

I will open a PR for this now

[bennahugo]

Ok I have a solution for you if you want to use the system casacore versions as you have done. Please apply the patch for that to work. The fix here is to downgrade to python-casacore==3.0.0 AFTER installing all the meqtrees python packages, this ensures you don't have mismatching ABI versions for casacore libraries.

There is something very broken in the multi-python installation on jake. The 3.6 libraries no longer binds correctly for casacore, hence it won't compile properly against this and the boost libraries are too old to support 3.8. Hence why I don't like having multiple non-default python versions on the same system path @o-smirnov

I have tested that this solution works on a clean 3.6 virtualenv following the instructions I have on the wiki - the testsuite for MT runs through.

Edit: I have released v1.9.0.1 for timba to include the patch mentioned

[bennahugo]

Note this may or may not solve the original pyxis problem --- pyxis tests runs through for me but your script may highlight an issue not caught by the test suite. At least it will be reproducable in a python3 environment where a fix can be made @o-smirnov .

[bennahugo]

Ok I can at least confirm there is a compilation issue that is not picked up with the ubuntu 20.04 build test. I will make a PR to fix this. However there is also an issue when building with older casacore libraries are present on the system which I'm looking into to fix

On Wed, Jun 8, 2022 at 12:03 PM Oleg Smirnov @.***> wrote:

Yep, that's why I want @kamva1 https://github.com/kamva1's command sequence. It seems the 2.7 KERN versions were pottering along ok, until I started updating the system.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/meqtrees/issues/887#issuecomment-1149718197, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6QVRAT46Z5BMS7TZMLVOBVY3ANCNFSM5YDZHJFQ . You are receiving this because you were mentioned.Message ID: @.***>

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town