losoto error

[HannoSpreeuw]

[hspreeuw@fs5 ~]$ git clone git@github.com:EOSC-LOFAR/prefactor-cwl.git Cloning into 'prefactor-cwl'... remote: Enumerating objects: 62, done. remote: Counting objects: 100% (62/62), done. remote: Compressing objects: 100% (40/40), done. remote: Total 714 (delta 25), reused 40 (delta 19), pack-reused 652 Receiving objects: 100% (714/714), 7.47 MiB | 1.27 MiB/s, done. Resolving deltas: 100% (449/449), done. [hspreeuw@fs5 ~]$ cd prefactor-cwl/ [hspreeuw@fs5 prefactor-cwl]$ make run-singularity virtualenv -p python2 .virtualenv Running virtualenv with interpreter /home/hspreeuw/miniconda2/bin/python2 New python executable in .virtualenv/bin/python2 Also creating executable in .virtualenv/bin/python ERROR: The executable .virtualenv/bin/python2 is not functioning ERROR: It thinks sys.prefix is u'/usr' (should be u'/home/hspreeuw/prefactor-cwl/.virtualenv') ERROR: virtualenv is not compatible with this system or executable make: *** [.virtualenv/] Error 100 [hspreeuw@fs5 prefactor-cwl]$ which pip ~/miniconda3/bin/pip [hspreeuw@fs5 prefactor-cwl]$ which pip2 ~/miniconda2/bin/pip2 [hspreeuw@fs5 prefactor-cwl]$ which pip3 ~/miniconda3/bin/pip3 [hspreeuw@fs5 prefactor-cwl]$ which python ~/miniconda3/bin/python [hspreeuw@fs5 prefactor-cwl]$ which python2 ~/miniconda2/bin/python2 [hspreeuw@fs5 prefactor-cwl]$ which python3 ~/miniconda3/bin/python3

[mr-c]

pip show virtualenv ?

Can you turn off your conda installation?

If that doesn't work, I would try the following modification, to use the system python.

rm -Rf .virtualenv

diff --git a/Makefile b/Makefile
index eeb9d90..d16d650 100644
--- a/Makefile
+++ b/Makefile
@@ -13,7 +13,7 @@ SMALL_ARCHIVE=L570745_uv_first10.MS.tar.xz

 .virtualenv/:
-       virtualenv -p python2 .virtualenv
+       /usr/bin/virtualenv -p /usr/bin/python2 .virtualenv

 .virtualenv/bin/cwltool: .virtualenv/
        .virtualenv/bin/pip install -r requirements.txt

[HannoSpreeuw]

[hspreeuw@fs5 prefactor-cwl]$ pip show virtualenv Traceback (most recent call last): File "/home/hspreeuw/miniconda3/bin/pip", line 7, in from pip._internal import main File "/usr/lib/python2.7/site-packages/pip/init.py", line 15, in from pip.vcs import git, mercurial, subversion, bazaar # noqa File "/usr/lib/python2.7/site-packages/pip/vcs/subversion.py", line 9, in from pip.index import Link File "/usr/lib/python2.7/site-packages/pip/index.py", line 30, in from pip.wheel import Wheel, wheel_ext File "/usr/lib/python2.7/site-packages/pip/wheel.py", line 6, in import compileall File "/home/hspreeuw/miniconda3/lib/python3.7/compileall.py", line 20, in from concurrent.futures import ProcessPoolExecutor File "/usr/lib/python2.7/site-packages/concurrent/futures/init.py", line 8, in from concurrent.futures._base import (FIRST_COMPLETED, File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 357 raise type(self._exception), self._exception, self._traceback ^ SyntaxError: invalid syntax

[HannoSpreeuw]

I removed both miniconda2 and miniconda3 from my PATH. Now I get: [hspreeuw@fs5 ~]$ git clone git@github.com:EOSC-LOFAR/prefactor-cwl.git Cloning into 'prefactor-cwl'... remote: Enumerating objects: 62, done. remote: Counting objects: 100% (62/62), done. remote: Compressing objects: 100% (40/40), done. remote: Total 714 (delta 25), reused 40 (delta 19), pack-reused 652 Receiving objects: 100% (714/714), 7.47 MiB | 903.00 KiB/s, done. Resolving deltas: 100% (449/449), done. [hspreeuw@fs5 ~]$ cd prefactor-cwl/ [hspreeuw@fs5 prefactor-cwl]$ make run-singularity virtualenv -p python2 .virtualenv Running virtualenv with interpreter /usr/bin/python2 New python executable in .virtualenv/bin/python2 Also creating executable in .virtualenv/bin/python Installing Setuptools..............................................................................................................................................................................................................................done. Installing Pip.....................................................................................................................................................................................................................................................................................................................................done. .virtualenv/bin/pip install -r requirements.txt Downloading/unpacking html5lib (from -r requirements.txt (line 1)) Downloading html5lib-1.0.1.tar.gz (252kB): 252kB downloaded Running setup.py egg_info for package html5lib Traceback (most recent call last): File "", line 16, in File "/home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib/setup.py", line 54, in MarkerEvaluation = pkg_resources.MarkerEvaluation AttributeError: 'module' object has no attribute 'MarkerEvaluation' Complete output from command python setup.py egg_info: Traceback (most recent call last):

File "", line 16, in

File "/home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib/setup.py", line 54, in

MarkerEvaluation = pkg_resources.MarkerEvaluation

AttributeError: 'module' object has no attribute 'MarkerEvaluation'

---

Cleaning up... Command python setup.py egg_info failed with error code 1 in /home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib Storing complete log in /home/hspreeuw/.pip/pip.log make: *** [.virtualenv/bin/cwltool] Error 1

[mr-c]

.virtualenv/bin/pip install -U pip
.virtualenv/bin/pip install -U setuptools wheel

[HannoSpreeuw]

Excellent. Thanks. It ran some tens of minutes and I expected it to run into a known issue like #23. This is with module load singularity/2.5.2 However, it ended with

Output is missing expected field file:///home/hspreeuw/prefactor-cwl/prefactor.cwl#h5imp_cal/losoto_h5

More verbose ouptput:

DEBUG - Writing visibilities to column: CORRECTED_DATA
DEBUG - Read time (meta data): timer: avg = 29.1 us, total =  146 us, count =         5
DEBUG - Write time: timer: avg = 1.59 ms, total = 9.57 ms, count =         6
DEBUG - Executing a Finalize command:
Command: Finalize
INFO - bbs-reducer terminated successfully.
INFO - clone CA                 : avg =  116 ns, total =  250 ms, count =   2158792
INFO - new CA                   : avg = 47.3 ns, total =  109 ms, count =   2299856
[job calib_cal_20] completed success
[step calib_cal] completed success
[workflow ] starting step h5imp_cal
[step h5imp_cal] start
[job h5imp_cal] Output of job will be cached in /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8
Using local copy of Singularity image found in /home/hspreeuw/prefactor-cwl
[job h5imp_cal] /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8:/var/spool/cwl:rw \
    --bind \
    /tmp/tmps8BRRE:/tmp:rw \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/3624b3f46f1ad175ef15dc5eba8a8f4e/calibrated.MS:/var/lib/cwl/stg11a2c476-603e-4ed2-834e-40e24bbde677/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/b1093dc6bb04e71d4bad0af1ee108706/calibrated.MS:/var/lib/cwl/stg0ed8e3ba-7f9f-40d6-91e5-d81beffc54ea/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/3cd85532f2f244d2d5191f5ab15628c5/calibrated.MS:/var/lib/cwl/stg6efb7e22-4da8-4dbc-820e-47548ca0b38c/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/20bac55ad3273acd3ce5b7dabf81abde/calibrated.MS:/var/lib/cwl/stg8e162382-352c-493e-bc3d-c88c2aa64928/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/bcee14698a197ead0d0c41d881ce4ea7/calibrated.MS:/var/lib/cwl/stg95a515c6-dadd-4943-8b18-6b1a426cf863/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/95502154f927525857708faef4477b85/calibrated.MS:/var/lib/cwl/stge1d501bf-f860-489b-a122-8ef85d5927ce/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/aea5c783e503950d7b2b834cc147648d/calibrated.MS:/var/lib/cwl/stge3c394af-8ca2-4416-9f47-a23ffbe0692e/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/c3cf20bd9ddb627f75c586d7da8035d8/calibrated.MS:/var/lib/cwl/stg11cdf87f-e9f6-4c4d-8bb4-feff442eede1/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/2ef9ae2e321c31377d428907f2b5a8d4/calibrated.MS:/var/lib/cwl/stg0ab7482d-b892-46b8-855f-a06643a74279/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/91524ca1bd4c45c354838529554b5edf/calibrated.MS:/var/lib/cwl/stgd54b5aec-b4d7-4581-901b-4e259f997dfe/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/bcc318d76575432116bec4a8ea99a747/calibrated.MS:/var/lib/cwl/stg0356c231-a49e-4016-8b13-c509fc854987/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/64f066e64c96fb48e28ae523dbeb4bbc/calibrated.MS:/var/lib/cwl/stg4333efde-286a-4078-9fe9-30ebf189de9c/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/1cef29b2f4600c7a9c0b8cc90507126e/calibrated.MS:/var/lib/cwl/stg8aaaacfb-b8eb-414d-8446-9f06cb80bb04/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/fa5b9b4c52081dc0e3bffae6056f4370/calibrated.MS:/var/lib/cwl/stg68f86e9a-b504-4cfa-a710-60c1f6f0c5a2/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/0ad9c6962ffdf5565c81a1969b5f3d82/calibrated.MS:/var/lib/cwl/stg58d192ef-145d-44c8-bb2c-fbe7a0ab8ae5/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/24f250d71a06f473bedcaa31222e4359/calibrated.MS:/var/lib/cwl/stg5f7d8bd7-c898-47b0-8ac7-2a00d4ae1438/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/31ac1f24d373fc2220cecc7a7d7a8947/calibrated.MS:/var/lib/cwl/stg4e17ff09-2f0f-42b5-bec9-e239e8f2586e/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/72f5ef74dc0fbb161e5dec73019f144d/calibrated.MS:/var/lib/cwl/stg365999d8-1036-485d-b67f-3dd264d93991/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/5c642df6490546980cf6ff08b767f036/calibrated.MS:/var/lib/cwl/stg4d8f9ea6-d802-4129-9e61-f83645eb5788/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/a80f038dbebd6bf9d681bad15157b547/calibrated.MS:/var/lib/cwl/stg7add5ebb-ba24-4f36-b93a-63f9df208b55/calibrated.MS:ro \
    --pwd \
    /var/spool/cwl \
    /home/hspreeuw/prefactor-cwl/kernsuite-prefactor.img \
    python \
    /usr/lib/prefactor/scripts/losotoImporter.py \
    losoto.h5 \
    -c \
    7 \
    -s \
    sol000 \
    /var/lib/cwl/stg8aaaacfb-b8eb-414d-8446-9f06cb80bb04/calibrated.MS \
    /var/lib/cwl/stg0ab7482d-b892-46b8-855f-a06643a74279/calibrated.MS \
    /var/lib/cwl/stg8e162382-352c-493e-bc3d-c88c2aa64928/calibrated.MS \
    /var/lib/cwl/stg4333efde-286a-4078-9fe9-30ebf189de9c/calibrated.MS \
    /var/lib/cwl/stg68f86e9a-b504-4cfa-a710-60c1f6f0c5a2/calibrated.MS \
    /var/lib/cwl/stg11cdf87f-e9f6-4c4d-8bb4-feff442eede1/calibrated.MS \
    /var/lib/cwl/stg7add5ebb-ba24-4f36-b93a-63f9df208b55/calibrated.MS \
    /var/lib/cwl/stg4d8f9ea6-d802-4129-9e61-f83645eb5788/calibrated.MS \
    /var/lib/cwl/stg0ed8e3ba-7f9f-40d6-91e5-d81beffc54ea/calibrated.MS \
    /var/lib/cwl/stg0356c231-a49e-4016-8b13-c509fc854987/calibrated.MS \
    /var/lib/cwl/stg6efb7e22-4da8-4dbc-820e-47548ca0b38c/calibrated.MS \
    /var/lib/cwl/stg4e17ff09-2f0f-42b5-bec9-e239e8f2586e/calibrated.MS \
    /var/lib/cwl/stg365999d8-1036-485d-b67f-3dd264d93991/calibrated.MS \
    /var/lib/cwl/stg5f7d8bd7-c898-47b0-8ac7-2a00d4ae1438/calibrated.MS \
    /var/lib/cwl/stg11a2c476-603e-4ed2-834e-40e24bbde677/calibrated.MS \
    /var/lib/cwl/stg58d192ef-145d-44c8-bb2c-fbe7a0ab8ae5/calibrated.MS \
    /var/lib/cwl/stgd54b5aec-b4d7-4581-901b-4e259f997dfe/calibrated.MS \
    /var/lib/cwl/stge1d501bf-f860-489b-a122-8ef85d5927ce/calibrated.MS \
    /var/lib/cwl/stg95a515c6-dadd-4943-8b18-6b1a426cf863/calibrated.MS \
    /var/lib/cwl/stge3c394af-8ca2-4416-9f47-a23ffbe0692e/calibrated.MS
/usr/lib/python2.7/dist-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
/usr/lib/python2.7/dist-packages/lofar/parmdb/__init__.py:22: RuntimeWarning: to-Python converter for std::vector<double, std::allocator<double> > already registered; second conversion method ignored.
  from _parmdb import ParmDB
/usr/lib/python2.7/dist-packages/lofar/parmdb/__init__.py:22: RuntimeWarning: to-Python converter for std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > already registered; second conversion method ignored.
  from _parmdb import ParmDB
Traceback (most recent call last):
  File "/usr/lib/prefactor/scripts/losotoImporter.py", line 16, in <module>
    from losoto.importer import create_h5parm
ImportError: No module named importer
[job h5imp_cal] Job error:
Error collecting output for parameter 'losoto_h5':
steps/h5imp_cal.cwl:48:7: Did not find output file with glob pattern: '[u'losoto.h5']'
[job h5imp_cal] completed permanentFail
[step h5imp_cal] Output is missing expected field file:///home/hspreeuw/prefactor-cwl/prefactor.cwl#h5imp_cal/losoto_h5
[step h5imp_cal] completed permanentFail
[workflow ] completed permanentFail
{
    "psf": null, 
    "losoto_h5": null, 
    "dtec_allsols": null, 
    "dclock_1st": null, 
    "phase_array": null, 
    "amplitude_array": null, 
    "phase_xx_yy_offset": null, 
    "amp_allsols": null, 
    "dclock_allsols": null, 
    "freqs_for_phase_array": null, 
    "dclock_1st_sm": null, 
    "residual": null, 
    "polYY_dirpointing": null, 
    "dirty": null, 
    "model": null, 
    "station_names": null, 
    "dTEC_1st": null, 
    "polXX_dirpointing": null, 
    "dTEC_1st_sm": null
}
Final process status is permanentFail
make: *** [run-singularity] Error 1

[mr-c]

@HannoSpreeuw I'm glad that worked. I think the key error is

from losoto.importer import create_h5parm
ImportError: No module named importer

[HannoSpreeuw]

Is that someting we can fix using pip?

[mr-c]

@HannoSpreeuw that a problem with the container or something within it

[mr-c]

You can request a fix at https://github.com/kernsuite/packaging/issues

[HannoSpreeuw]

But why kernsuite? Couldn't it be prefactor? Since we have

File "/usr/lib/prefactor/scripts/losotoImporter.py", line 16, in

[mr-c]

That's a path within the container. It is true, could be a prefactor bug.

[HannoSpreeuw]

@tammojan please comment.

[gijzelaerr]

KERN-dev uses losoto 2.0, it looks like prefactor is not compatible with this release?

[gijzelaerr]

KERN-3 still contains the old losoto, trying that one out now.