Segmentation Fault running DynSpecMS

[AntoniaR]

Running ms2dynspec.py I get a segmentation fault.

As far as I can tell, everything is installed correctly but maybe I am doing something wrong in running it. I am running it within a singularity container for DDFacet. My data is calibrated using LINC and I have subtracted the skymodel from the visibilities to give the column “SUBTRACTED_DATA”. I can share a measurement set for debugging if useful.

This is my command and the script output:

Singularity> python3 ms2dynspec.py --ms ~/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms --data "SUBTRACTED_DATA" --srclist ~/GRB201006A/GRB201006A_src.csv
 ______             _____                ___  ___ _____  
 |  _  \           /  ___|               |  \/  |/  ___| 
 | | | |_   _ _ __ \ `--. _ __   ___  ___| .  . |\ `--.  
 | | | | | | | '_ \ `--. \ '_ \ / _ \/ __| |\/| | `--. \ 
 | |/ /| |_| | | | /\__/ / |_) |  __/ (__| |  | |/\__/ / 
 |___/  \__, |_| |_\____/| .__/ \___|\___\_|  |_/\____/  
         __/ |           | |                             
        |___/            |_|                             
                                 version v1.0-29-g0ad33b5
 - 08:10:14 - AsyncProcessPool             | Fixing parent process to vthread 0
 - 08:10:14 - AsyncProcessPool             | Worker affinities not specified, leaving unset
 - 08:10:14 - ClassMS                      | found 1 MSs matching /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: selecting DDID 0
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: selecting field 0
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: non-default column DATA
 - 08:10:14 - ClassMS                      | 1 MS section(s) selected
 - 08:10:14 - DynSpecMS                    | DynSpecMS version v1.0-29-g0ad33b5 starting up
Segmentation fault (core dumped)

I’m meant to be passing the SUBTRACTED_DATA column but this doesn’t seem to be passing through as I think the script is using “DATA”.

My source list is (file attached):

GRB201006A 61.89270 65.16462
BG1 61.542444 65.224421
BG2 62.172479 65.253684
BG3 62.120941 65.059329

The positions are in degrees and I give 3 background regions. My source (GRB201006A) is at the centre of the field. GRB201006A_src.csv

[cyriltasse]

Hey @AntoniaR - can you try --data SUBTRACTED_DATA instead of --data "SUBTRACTED_DATA", could be just that actually...

[AntoniaR]

Unfortunately not, I tried that first and then added in the speech marks when it didn't work.

[cyriltasse]

It's weird that it segfaults anyway - and that it doesn't take the arguments you're giving it... can you launch it without the python3, and if that fails can you put your singularity image somewhere together with the ms and the source list so I can reproduce it myself?

[AntoniaR]

Sadly, that didn't work either. Here is the measurement set:

L796450_121MHz_uv_pre-cal.ms.zip

I will e-mail a link to my DDFacet image to you directly.

[AntoniaR]

Hi! I've tried digging into this further and identified where the problem is coming from.

Here, the MS is being opened with the following command: https://github.com/cyriltasse/DynSpecMS/blob/master/ClassDynSpecMS.py#L270 t0 = table(MSName, ack=False)

Firstly, it is using pyrap which I believe should now be updated to casacore (from casacore.tables import table). Secondly, when I run this command outside of the script, I get the following error:

Singularity> ipython 
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.26.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from casacore.tables import table                                                          

In [2]: table('/home/rowlinson/GRB201006A/target/plotdata/L796450_166MHz_uv_pre-cal.ms')           
free(): invalid pointer
Aborted (core dumped)

So I think it is something to do with memory allocations (see e.g. https://stackoverflow.com/questions/56346569/free-invalid-pointer-aborted-core-dumped ) but I don't know how to fix it. I guess it's something to do with using a ddf image.

[o-smirnov]

I believe pyrap is just a plain alias for casacore these days, to preserve backwards compatibility, so it shouldn't be causing issues per se. Unless the image somehow has truly ancient pyrap libraries installed? Where's the image from @cyriltasse @AntoniaR and how is it made, any idea?

I have seen errors like this when the casacore binary install (i.e. libraries under /usr/lib) version doesn't match the Python libraries, so my money's on that.

[mhardcastle]

You're sure the measurement set is not simply corrupt?

[AntoniaR]

@o-smirnov The image is one I got from my student who has been using DDFacet successfully. I'm not sure what versions are installed or how it was made - I think she got it from the Leiden team.

@mhardcastle The measurement table is opened fine when I am outside of the DDFacet image (see below), plus it images fine. So I'm confident that the measurement set is fine.

(pythonEnv) rowlinson@lofproc:~/GRB201006A/target/plotdata$ python
Python 3.8.13 (default, Apr 19 2022, 00:53:22) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from casacore.tables import table
>>> table('L796450_166MHz_uv_pre-cal.ms')
Successful readonly open of default-locked table L796450_166MHz_uv_pre-cal.ms: 26 columns, 12726300 rows
<casacore.tables.table.table object at 0x7f12dd0dd180>

As it is likely my DDFacet image, does anyone have a recent one that I could try out?

[mhardcastle]

Try https://uhhpc.herts.ac.uk/~mjh/ddf_py3_d11_ctmaster.sif ?

[AntoniaR]

@mhardcastle Thank you! This DDFacet image got past my segmentation fault error so I think this issue can be closed. I have a different bug to track down now :)

[mhardcastle]

I'll close the issue (I'll remove the web link too because I don't want people relying on it -- but I always have a working DDFacet singularity for future reference).

[AntoniaR]

@mhardcastle Excellent!

Thank you all very much for the help getting this running!