TDL job "simulate" crashing

TariqBlecher commented 9 years ago

Have there been changes made to the simulator in the last few weeks which might cause this error when running a simulation job as I don't think my script has changed..

TDL script successfully compiled. 215 node definitions

(of which 2 are root nodes) sent to meqserver.

Running TDL job "simulate"

Job result: None

No more commands

meqserver reported 1 error(s) during the run:

000: node 'VisDataMux': execute() failed: TableMeasRefDesc error: old refcode Undefined does not exist anymore (return code 0x810021)

Stopping the meqserver

o-smirnov commented 9 years ago

What's the provenance of the MS? I've seen this error with MSs made with CASA 4.3+. @rdeane knows all about that pain.

Current solution is to avoid CASA 4.3 and higher, but I'll try to find a better one with @gijzelaerr tomorrow.

TariqBlecher commented 9 years ago

I (blecher@jake) own the MS, the MS was made with makems

On Sun, Jul 26, 2015 at 7:32 PM, Oleg Smirnov notifications@github.com wrote:

What's the provenance of the MS? I've seen this error with MSs made with CASA 4.3+. @rdeane https://github.com/rdeane knows all about that pain.

Current solution is to avoid CASA 4.3 and higher, but I'll try to find a better one with @gijzelaerr https://github.com/gijzelaerr tomorrow.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125020109 .

o-smirnov commented 9 years ago

Hmmm that's odd. Will take a look with @gijzelaerr tomorrow. Probably related to the current casacore upgrades...

TariqBlecher commented 9 years ago

in the meantime, I'm trying to use im.argo.predict_vis but the predict_vis does not seem to be a module in im.argo anymore?

----> 1 im.argo.predict_vis

AttributeError: 'module' object has no attribute 'predict_vis'

On Sun, Jul 26, 2015 at 7:38 PM, Oleg Smirnov notifications@github.com wrote:

Hmmm that's odd. Will take a look with @gijzelaerr https://github.com/gijzelaerr tomorrow. Probably related to the current casacore upgrades...

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125021322 .

o-smirnov commented 9 years ago

It's im.lwimager.predict_vis for now...

On Sun, Jul 26, 2015 at 7:41 PM Tariq Blecher notifications@github.com wrote:

in the meantime, I'm trying to use im.argo.predict_vis but the predict_vis does not seem to be a module in im.argo anymore?

----> 1 im.argo.predict_vis

AttributeError: 'module' object has no attribute 'predict_vis'

On Sun, Jul 26, 2015 at 7:38 PM, Oleg Smirnov notifications@github.com wrote:

Hmmm that's odd. Will take a look with @gijzelaerr https://github.com/gijzelaerr tomorrow. Probably related to the current casacore upgrades...

— Reply to this email directly or view it on GitHub < https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125021322

.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022012 .

TariqBlecher commented 9 years ago

Yeah there is definitely something strange happening with the casacore tables..Pyrap can't even access the Station Names.

----> 1 t.getcol('NAME')

/usr/lib/python2.7/dist-packages/pyrap/tables/table.pyc in getcol(self, columnname, startrow, nrow, rowincr) 854 # i = inx* 855 # except: --> 856 return self._getcol (columnname, startrow, nrow, rowincr) 857 858 def getvarcol (self, columnname, startrow=0, nrow=-1, rowincr=1):

RuntimeError: PycValueHolder: unknown casa data type Char

On Sun, Jul 26, 2015 at 7:44 PM, Oleg Smirnov notifications@github.com wrote:

It's im.lwimager.predict_vis for now...

On Sun, Jul 26, 2015 at 7:41 PM Tariq Blecher notifications@github.com wrote:

in the meantime, I'm trying to use im.argo.predict_vis but the predict_vis does not seem to be a module in im.argo anymore?

----> 1 im.argo.predict_vis

AttributeError: 'module' object has no attribute 'predict_vis'

On Sun, Jul 26, 2015 at 7:38 PM, Oleg Smirnov notifications@github.com wrote:

Hmmm that's odd. Will take a look with @gijzelaerr https://github.com/gijzelaerr tomorrow. Probably related to the current casacore upgrades...

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125021322

.

— Reply to this email directly or view it on GitHub < https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022012

.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022203 .

o-smirnov commented 9 years ago

Tariq, could you please post a link to the MS and to the makems config file used to create it? We'd like to chase down this error.

On Sun, Jul 26, 2015 at 7:54 PM Tariq Blecher notifications@github.com wrote:

Yeah there is definitely something strange happening with the casacore tables..Pyrap can't even access the Station Names.

----> 1 t.getcol('NAME')

/usr/lib/python2.7/dist-packages/pyrap/tables/table.pyc in getcol(self, columnname, startrow, nrow, rowincr) 854 # i = inx* 855 # except: --> 856 return self._getcol (columnname, startrow, nrow, rowincr) 857 858 def getvarcol (self, columnname, startrow=0, nrow=-1, rowincr=1):

RuntimeError: PycValueHolder: unknown casa data type Char

On Sun, Jul 26, 2015 at 7:44 PM, Oleg Smirnov notifications@github.com wrote:

It's im.lwimager.predict_vis for now...

On Sun, Jul 26, 2015 at 7:41 PM Tariq Blecher notifications@github.com wrote:

in the meantime, I'm trying to use im.argo.predict_vis but the predict_vis does not seem to be a module in im.argo anymore?

----> 1 im.argo.predict_vis

AttributeError: 'module' object has no attribute 'predict_vis'

On Sun, Jul 26, 2015 at 7:38 PM, Oleg Smirnov < notifications@github.com> wrote:

Hmmm that's odd. Will take a look with @gijzelaerr https://github.com/gijzelaerr tomorrow. Probably related to the current casacore upgrades...

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125021322

.

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022012

.

— Reply to this email directly or view it on GitHub < https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022203

.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022899 .

TariqBlecher commented 9 years ago

So it can be found on jake: /home/blecher/MeqSilhouette/src/mean30.MS & mean30.MS.cfg

You can ignore the last issue concerning the Station Names, that was because pyxis was not parsing the pyxis-MeqSilhouette.conf file properly.

On Mon, Jul 27, 2015 at 1:46 PM, Oleg Smirnov notifications@github.com wrote:

Tariq, could you please post a link to the MS and to the makems config file used to create it? We'd like to chase down this error.

On Sun, Jul 26, 2015 at 7:54 PM Tariq Blecher notifications@github.com

wrote:

Yeah there is definitely something strange happening with the casacore tables..Pyrap can't even access the Station Names.

----> 1 t.getcol('NAME')

/usr/lib/python2.7/dist-packages/pyrap/tables/table.pyc in getcol(self, columnname, startrow, nrow, rowincr) 854 # i = inx* 855 # except: --> 856 return self._getcol (columnname, startrow, nrow, rowincr) 857 858 def getvarcol (self, columnname, startrow=0, nrow=-1, rowincr=1):

RuntimeError: PycValueHolder: unknown casa data type Char

On Sun, Jul 26, 2015 at 7:44 PM, Oleg Smirnov notifications@github.com wrote:

It's im.lwimager.predict_vis for now...

On Sun, Jul 26, 2015 at 7:41 PM Tariq Blecher < notifications@github.com> wrote:

in the meantime, I'm trying to use im.argo.predict_vis but the predict_vis does not seem to be a module in im.argo anymore?

----> 1 im.argo.predict_vis

AttributeError: 'module' object has no attribute 'predict_vis'

On Sun, Jul 26, 2015 at 7:38 PM, Oleg Smirnov < notifications@github.com> wrote:

Hmmm that's odd. Will take a look with @gijzelaerr https://github.com/gijzelaerr tomorrow. Probably related to the current casacore upgrades...

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125021322

.

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022012

.

— Reply to this email directly or view it on GitHub <

https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022203

.

— Reply to this email directly or view it on GitHub < https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125022899

.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/34#issuecomment-125176270 .

IanHeywood commented 9 years ago

Kumar Golap has implemented a trick for banishing this error:

2015-08-16 23:32:24 SEVERE  split::ms::split    Exception Reported: TableMeasRefDesc error: old refcode Undefined does not exist anymore

2015-08-16 23:32:24 SEVERE  split::::casa   *** Error: TableMeasRefDesc error: old refcode Undefined does not exist anymore

I've been using CASA 4.1 which was refusing to deal with some post-4.3 Measurement Sets. You'll need CASA 4.3 to correct the problem (4.4 might work but I've only tried 4.3), after which both 4.3 and 4.1 were happy to read the MS. Just start CASA 4.3 and run the following command on your problematic MS:

!fixspwbackport mydata.ms

complete with the leading exclamation mark.

Details can be found here.

jfunction commented 8 years ago

Hi - I am having this issue on CASA 4.2.2 which is what https://github.com/radio-astro/docker-casapy/ is using. I have also installed lwimager and casacore libs via the radioastro ppas. The problem still persists.

Specifically I am following the pyxis tutorial (https://github.com/ratt-ru/ratt-interferometry-course/tree/master/practicals/pyxis-101) which makes a measurement set with makems. I thought that this problem was only meant to occur with CASA > 4.3 ?

Do I need to install CASA 4.3 in order to fix this issue?

o-smirnov commented 8 years ago

OK this is not a CASA issue truly, this is a casacore issue. Here's a minimal recipe for triggering it:

makems makems_kat7.cfg
lwimager ms=kat7_4h60s.MS_p0

but, it has to be lwimager from the lwimager package (casacore1-linked). lwimager from the newer casarest package (casacore2-linked) works fine.

So I suppose the real issue is, casacore1-based software does not recognize MSs done by casacore2-based software. I'll investigate a little to see if the fixspwbackport magic above can be recreated with pyrap.

o-smirnov commented 8 years ago

OK, the following bit of Python can be used to make an MS palatable if this error reoccurs. I need to insert this into Pyxis.

from pyrap.tables import table
tab=table("kat7_4h60s.MS/SPECTRAL_WINDOW/",readonly=False)
tab.getcolkeyword("CHAN_FREQ","MEASINFO")
a=tab.getcolkeyword("CHAN_FREQ","MEASINFO")
a['TabRefCodes'] = a['TabRefCodes'][:-1]
a['TabRefTypes'] = a['TabRefTypes'][:-1]
a=tab.putcolkeyword("CHAN_FREQ","MEASINFO",a)
a=tab.getcolkeyword("REF_FREQUENCY","MEASINFO")
a['TabRefTypes'] = a['TabRefTypes'][:-1]
a['TabRefCodes'] = a['TabRefCodes'][:-1]
a=tab.putcolkeyword("REF_FREQUENCY","MEASINFO",a)
tab.close()

ratt-ru / meqtrees-cattery