search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

wrong phase rotation in casa fixvis: add chgcentre to stimela? #337

Closed paoloserra closed 5 years ago

paoloserra commented 5 years ago

@SpheMakh Could you include Offringa's chgcentre in Stimela?

In order to process ASKAP data we need to rotate the phases of the target. I've implemented this in MeerKATHI's split_target worker using Casa fixvis. Unfortunately, fixvis uses the small angle approximation and does not work properly for large offsets. A change of phase centre by ~1 deg leads to several arcmin error in source position. @dane-kleiner and I have tested this on both ASKAP and MeerKAT data.

Unless somebody has a better idea I suggest we use Offringa's chgcentre instead of fixvis. Hence my request to include chgcentre in Stimela. (I've checked and I don't think it's there yet.)

Thanks.

SpheMakh commented 5 years ago

Yes, I will add chgcentre in stimela

https://github.com/SpheMakh/Stimela/issues/307

KshitijT commented 5 years ago

Check out for usage: https://sourceforge.net/p/wsclean/wiki/chgcentre/

chgcentre myset.ms 09h18m05.8s -12d05m44s
SpheMakh commented 5 years ago

PR has been submitted to Stimela. It should go through in a couple of hours https://github.com/SpheMakh/Stimela/pull/311#pullrequestreview-183158644

@dane-kleiner can then add it to the pipeline

SpheMakh commented 5 years ago

The PR has gone through. If you update stimela, then chgcentre should be available as a stimela cab

pip install -U https://github.com/SpheMakh/Stimela # to update
paoloserra commented 5 years ago

@SpheMakh I've implemented this in MeerKATHI. However, when I run it I get

meerkathi - 2019-03-14 18:40:09,966 INFO - Running: docker start -a changecentre_0-139791542130200155258520936
meerkathi - 2019-03-14 18:40:10,270 INFO - Running: chgcentre /home/pserra/msdir/ngc1399_20MHz-chgcen.ms 03h37m34.611s -35d19m25.64s
meerkathi - 2019-03-14 18:40:10,470 INFO - Illegal instruction (core dumped)
meerkathi - 2019-03-14 18:40:10,871 INFO - Traceback (most recent call last):
meerkathi - 2019-03-14 18:40:11,071 INFO -   File "/scratch/code/run.py", line 27, in <module>
meerkathi - 2019-03-14 18:40:11,172 CRITICAL - INFO:STIMELA-60:Recipe execution failed while running job changecentre_0

From shell the same command runs fine chgcentre ngc1399_20MHz-chgcen.ms 03h37m34.611s -35d19m25.64s

Any tips?

paoloserra commented 5 years ago

Note that I've tested this also outside MeerKATHI with a quick Stimela script. I get the same error.

SpheMakh commented 5 years ago

@Athanaseus any ideas?

Athanaseus commented 5 years ago

Hi @paoloserra I cannot reproduce this error. Is your stimela recipe as follows?

import  stimela

INPUT = 'input'
OUTPUT = 'output'
MSDIR = 'msdir'

stimela.register_globals()

recipe = stimela.Recipe('Change Centre', ms_dir=MSDIR)

recipe.add('cab/chgcentre', 'change_centre',
        {
         "msname": "ngc1399_20MHz-small.ms",
         "new-ra": "03h37m34.611s",
         "new-dec": "-35d19m25.64s",
        },
        input=INPUT,
        output=OUTPUT,
        label='chg_centre:: Centering')

recipe.run()

Output:


(meerkathi-venv) ramaila@stills:~/working/NGC_1399/run2$ stimela run chgcentre.py
2019-03-19 12:47:12,261 - STIMELA - INFO - ---------------------------------
2019-03-19 12:47:12,261 - STIMELA - INFO - Stimela version 1.0.1
2019-03-19 12:47:12,261 - STIMELA - INFO - Sphesihle Makhathini <sphemakh@gmail.com>
2019-03-19 12:47:12,261 - STIMELA - INFO - Running: Change Centre
2019-03-19 12:47:12,261 - STIMELA - INFO - ---------------------------------
2019-03-19 12:47:12,372 - STIMELA - INFO - Adding cab 'ramaila_cab/chgcentre' to recipe. The container will be named 'change_centre'
INFO:STIMELA:Adding cab 'ramaila_cab/chgcentre' to recipe. The container will be named 'change_centre'
2019-03-19 12:47:12,372 - STIMELA - INFO - Running job change_centre
INFO:STIMELA:Running job change_centre
2019-03-19 12:47:12,372 - STIMELA - INFO - STEP 1 :: chg_centre:: Centering
INFO:STIMELA:STEP 1 :: chg_centre:: Centering
INFO:root:Validating parameters...       CAB = chgcentre
INFO:root:Parameters validated and saved. Parameter file is: stimela_parameter_files/change_centre-140469320890888155299243234.json
2019-03-19 12:47:12,427 - STIMELA - INFO - Instantiating container [change_centre-140469320890888155299243234]. The container ID is printed below.
INFO:STIMELA:Instantiating container [change_centre-140469320890888155299243234]. The container ID is printed below.
Running: docker create --user 1124:1124  -v /home/ramaila/gitPackages/Stimela/stimela:/scratch/stimela:ro -v /home/ramaila/working/NGC_1399/run2/stimela_parameter_files:/configs:ro -v /home/ramaila/working/NGC_1399/run2/msdir:/home/ramaila/msdir:rw -v /home/ramaila/working/NGC_1399/run2/input:/input:ro -v /home/ramaila/working/NGC_1399/run2/output:/home/ramaila/output:rw  -e CONFIG=/configs/change_centre-140469320890888155299243234.json -e MSDIR=/home/ramaila/msdir -e INPUT=/input -e HOME=/home/ramaila/output -e OUTPUT=/home/ramaila/output -e LOGFILE=/home/ramaila/output/log-change_centre.txt  --name change_centre-140469320890888155299243234 --shm-size 1gb ramaila_cab/chgcentre
d922ce3fd3e11658b109edc0c1b3a2f2e8ddf6d2d11f1dcd9aa0ec0fbb9b8131

2019-03-19 12:47:13,509 - STIMELA - INFO - Starting container [change_centre-140469320890888155299243234]. Timeout set to -1. The container ID is printed below.
INFO:STIMELA:Starting container [change_centre-140469320890888155299243234]. Timeout set to -1. The container ID is printed below.
Running: docker start -a change_centre-140469320890888155299243234

Running: chgcentre /home/ramaila/msdir/ngc1399_20MHz-small.ms 03h37m34.611s -35d19m25.64s
A program to change the phase centre of a measurement set.
Written by André Offringa (offringa@gmail.com).

Processing field "NGC1399": 03h38m29.03s -35d27m02.4s -> 03h37m34.61s -35d19m25.64s (0.224209 deg)
Denormal shifting: 0,0 -> 0,0
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)  Requested data table IERSeop97 cannot be found in the searched directories:
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)+ /var/lib/casacore/data/ephemerides/
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)+ /var/lib/casacore/data/geodetic/
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)  Requested data table IERSpredict cannot be found in the searched directories:
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)+ /var/lib/casacore/data/ephemerides/
2019-03-19 10:47:15     WARN    MeasIERS::findTab (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasIERS.cc, line 381)+ /var/lib/casacore/data/geodetic/
2019-03-19 10:47:15     INFO    MeasIERS::fillMeas(MeasIERS::Files, Double)     Requested JD 58320 is outside the range of the IERS (Earth axis data) table.
2019-03-19 10:47:15     INFO    MeasIERS::fillMeas(MeasIERS::Files, Double) +   Calculations will proceed with less precision
2019-03-19 10:47:15     SEVERE  MeasTable::dUTC(Double) (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasTable.cc, line 4396)  Leap second table TAI_UTC seems out-of-date.
2019-03-19 10:47:15     SEVERE  MeasTable::dUTC(Double) (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasTable.cc, line 4396)+ Until the table is updated (see the CASA documentation or your system admin),
2019-03-19 10:47:15     SEVERE  MeasTable::dUTC(Double) (file /build/casacore-IDmUo1/casacore-2.4.1/measures/Measures/MeasTable.cc, line 4396)+ times and coordinates derived from UTC could be wrong by 1s or more.
Old uvw: [0, 0, 0] (0)
New [0, 0, 0] (0)

Old uvw: [0, 0, 0] (0)
New [0, 0, 0] (0)

Old uvw: [0, 0, 0] (0)
New [0, 0, 0] (0)

Old uvw: [0, 0, 0] (0)
New [0, 0, 0] (0)

Old uvw: [0, 0, 0] (0)
New [0, 0, 0] (0)

Changing phase centre: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%

2019-03-19 12:48:18,581 - STIMELA - INFO - Container [change_centre-140469320890888155299243234] has executed successfully
INFO:STIMELA:Container [change_centre-140469320890888155299243234] has executed successfully
2019-03-19 12:48:18,581 - STIMELA - INFO - Runtime was 0:01:05.150865.
INFO:STIMELA:Runtime was 0:01:05.150865.
2019-03-19 12:48:18,611 - STIMELA - INFO - Container change_centre-140469320890888155299243234 has been stopped.
INFO:STIMELA:Container change_centre-140469320890888155299243234 has been stopped.
Running: docker rm change_centre-140469320890888155299243234
change_centre-140469320890888155299243234

2019-03-19 12:48:19,643 - STIMELA - INFO - Saving pipeline information in .last_change_centre.json
INFO:STIMELA:Saving pipeline information in .last_change_centre.json
2019-03-19 12:48:19,644 - STIMELA - INFO - Recipe executed successfully
INFO:STIMELA:Recipe executed successfully
paoloserra commented 5 years ago

@Athanaseus my recipe was a little different, but I've tried with yours and get the same error.

(meerkathi-venv) pserra@corto:~/Astro/test_stimela$ stimela run changecentre.py 
2019-03-19 14:36:28,664 - STIMELA - INFO - ---------------------------------
2019-03-19 14:36:28,664 - STIMELA - INFO - Stimela version 1.0.1
2019-03-19 14:36:28,664 - STIMELA - INFO - Sphesihle Makhathini <sphemakh@gmail.com>
2019-03-19 14:36:28,664 - STIMELA - INFO - Running: Change Centre
2019-03-19 14:36:28,664 - STIMELA - INFO - ---------------------------------
2019-03-19 14:36:28,729 - STIMELA - INFO - Adding cab 'pserra_cab/chgcentre' to recipe. The container will be named 'change_centre'
INFO:STIMELA:Adding cab 'pserra_cab/chgcentre' to recipe. The container will be named 'change_centre'
2019-03-19 14:36:28,729 - STIMELA - INFO - Running job change_centre
INFO:STIMELA:Running job change_centre
2019-03-19 14:36:28,729 - STIMELA - INFO - STEP 1 :: chg_centre:: Centering
INFO:STIMELA:STEP 1 :: chg_centre:: Centering
INFO:root:Validating parameters...       CAB = chgcentre
INFO:root:Parameters validated and saved. Parameter file is: stimela_parameter_files/change_centre-140218406600864155300258872.json
2019-03-19 14:36:28,729 - STIMELA - INFO - Instantiating container [change_centre-140218406600864155300258872]. The container ID is printed below.
INFO:STIMELA:Instantiating container [change_centre-140218406600864155300258872]. The container ID is printed below.
Running: docker create --user 1000:1000  -v /home/pserra/Astro/meerkathi-venv/local/lib/python2.7/site-packages/stimela:/scratch/stimela:ro -v /home/pserra/Astro/test_stimela/stimela_parameter_files:/configs:ro -v /home/pserra/Astro/test_stimela/msdir:/home/pserra/msdir:rw -v /home/pserra/Astro/test_stimela/input:/input:ro -v /home/pserra/Astro/test_stimela/output:/home/pserra/output:rw  -e CONFIG=/configs/change_centre-140218406600864155300258872.json -e MSDIR=/home/pserra/msdir -e INPUT=/input -e HOME=/home/pserra/output -e OUTPUT=/home/pserra/output -e LOGFILE=/home/pserra/output/log-change_centre.txt  --name change_centre-140218406600864155300258872 --shm-size 1gb pserra_cab/chgcentre 
91272b28d391b128fedb51e546dd70a12e63a5675042e81ba1f7255062d7007f

2019-03-19 14:36:29,164 - STIMELA - INFO - Starting container [change_centre-140218406600864155300258872]. Timeout set to -1. The container ID is printed below.
INFO:STIMELA:Starting container [change_centre-140218406600864155300258872]. Timeout set to -1. The container ID is printed below.
Running: docker start -a change_centre-140218406600864155300258872
Running: chgcentre /home/pserra/msdir/ngc1399_20MHz-afr.ms 03h37m34.611s -35d19m25.64s

Illegal instruction (core dumped)
paoloserra commented 5 years ago

Do you think it's worth trying to update Stimela?

pip install --upgrade git+https://github.com/SpheMakh/Stimela
stimela build
Athanaseus commented 5 years ago

Yes @paoloserra let's give that a shot, coz I tested it on stills and seems to work fine.

paoloserra commented 5 years ago

Still the same

paoloserra commented 5 years ago

@Athanaseus Can you try after a Stimela update? Maybe you're using an older one?

Athanaseus commented 5 years ago

@paoloserra I'm running on the latest master (It still seems fine even after a new build). It's quite strange it throws a core dumped error. Can you please test it on a separate machine, just to confirm.

paoloserra commented 5 years ago

sure. @dane-kleiner ?

dane-kleiner commented 5 years ago

I'll test it today or tomorrow

dane-kleiner commented 5 years ago

Hey @Athanaseus,

I tested this as well and also got an error like @paoloserra. Below is the script and the output

import  stimela

INPUT = 'input'
OUTPUT = 'output'
MSDIR = 'msdir'

stimela.register_globals()

recipe = stimela.Recipe('Change Centre', ms_dir=MSDIR)

recipe.add('cab/chgcentre', 'change_centre',
        {
         "msname": "Fornax_B_beam04-corr.ms",
         "new-ra": "03h25m11.381s",
         "new-dec": "-34d03m44.46s",
        },
        input=INPUT,
        output=OUTPUT,
        label='chg_centre:: Centering')

recipe.run()
Running: docker start -a change_centre-140579236114192155316350545
Running: chgcentre /home/dkleiner/msdir/Fornax_B_beam04-corr.ms 03h25m11.381s -34d03m44.46s
Illegal instruction (core dumped)

Traceback (most recent call last):
  File "/scratch/code/run.py", line 27, in <module>
    utils.xrun(cab['binary'], args)
  File "/scratch/stimela/utils/__init__.py", line 110, in xrun
    raise StimelaCabRuntimeError('%s: returns errr code %d' % (command, process.returncode))
utils.StimelaCabRuntimeError: chgcentre: returns errr code 132
paoloserra commented 5 years ago

Yo, so I've tried all this on Stills and it works fine. Give me a sec to make one last attempt on my local machine.

paoloserra commented 5 years ago

No luck :(

I've tried with a fresh installation in a new venv following all steps at https://github.com/ska-sa/meerkathi#on-linux both on Stills and my local machine (after deleting ~/.stimela/*). On Stills the stimela script you posted worked fine. On my machine it resulted in the usual message

Running: chgcentre /home/pserra/msdir/ngc1399_20MHz-n13_t2_mst.ms 03h37m34.611s -35d19m25.64s
Illegal instruction (core dumped)
paoloserra commented 5 years ago

@Athanaseus would you be able to test this on a different machine? I'm running out of ideas...

Athanaseus commented 5 years ago

@paoloserra I managed to reproduce the error. I'll ping @SpheMakh to assist with debugging.

@SpheMakh should we perhaps try to install it from source in the container to see if the results are positive?

SpheMakh commented 5 years ago

@Athanaseus the kern package seems to be broken. I'll report this upstream, and try building from source in the meantime.

paoloserra commented 5 years ago

thanks guys

SpheMakh commented 5 years ago

The error is in chgcentre, and described in this wsclean ticket https://sourceforge.net/p/wsclean/tickets/50/

I have a fix in the current chgcentre stimela image (stimela/chgcentre:1.0.1). Please test if it works for you @paoloserra 

docker pull stimela/chgcentre:1.0.1
stimela build -uo chgcentre

But this will need a fix in from Andre; I'll open a ticket on sourceforge

SpheMakh commented 5 years ago

Keep track of the chgcentre ticket here https://sourceforge.net/p/wsclean/tickets/143/

SpheMakh commented 5 years ago

@gijzelaerr this also affects the kern chgcentre packages

gijzelaerr commented 5 years ago

please open an issue on the issue tracker on KERN when there is a new release or patch and I'll upload it to the KERN of your likings.

SpheMakh commented 5 years ago

Fixed in the current stimela.