search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

Gaintables cannot be written in Python 3 mode due to current casacore implementation issues #440

Open o-smirnov opened 3 years ago

o-smirnov commented 3 years ago

Are there still casacore issues blocking this?

caseyjlaw commented 3 years ago

I'd like to understand this issue. Can someone tag the other relevant discussions here? Also, is the recommended workaround to reinstall with the entire python2 stack?

o-smirnov commented 3 years ago

Well there was this: https://github.com/ratt-ru/CubiCal/issues/269

But I do have a nagging suspicion we can just disable the warning and let it write the damn tables and see if anything breaks. There was a nasty interregnum when python3-casacore had some teething issues, and that's when @bennahugo put this in. I believe the upstream issues have now been solved.

bennahugo commented 3 years ago

it has been an issue on my list. Unfortunately haven't got back to this yet as I'm trying to roll a release for killms at the moment. You can try removing the warnings, however it may be that you need to add UTF-8 encoding to the columns meant to contain casacore::string types like the antenna name column, etc. Unfortunately you cannot roll back to python 2.7. We no longer support it. In the interm you can always plot the cubical native paramdb tables from ipython?

bennahugo commented 3 years ago

An example is here: https://gist.github.com/bennahugo/921f931973b159b7f19348b46b38d836

caseyjlaw commented 3 years ago

Thanks for the quick feedback, all. I have commented out the lines that stop execution of saving calibration information. After rebuilding, I demonstrated that I can run cubical and that it saves paramdb files. I can read them into an ipython session and make plots of amplitude and phase of bandpasses. Is that sufficient to test this issue? Incidentally, I'm a bit confused as to why casacore is required to write these paramdb files. They seem to be python data structures, so how is casacore used?

JSKenyon commented 3 years ago

The paramdb objects can be written without casacore. The original issue was that we also support(ed) writing to casa gain tables. It is that functionality which broke due to upstream issues. I will take a look and see if the functionality can be restored now that things have settled down upstream.

In the meantime, using the paramdb objects as you seem to be doing will work just fine.

On Tue, 30 Mar 2021, 00:49 Casey Law, @.***> wrote:

Thanks for the quick feedback, all. I have commented out the lines that stop execution of saving calibration information. After rebuilding, I demonstrated that I can run cubical and that it saves paramdb files. I can read them into an ipython session and make plots of amplitude and phase of bandpasses. Is that sufficient to test this issue? Incidentally, I'm a bit confused as to why casacore is required to write these paramdb files. They seem to be python data structures, so what how is casacore used?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/440#issuecomment-809767234, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSHDWL5NPFU4SWY4QREQN3TGD7WHANCNFSM4YS74PZA .

o-smirnov commented 3 years ago

Yep, paramdb is the native format and it is always written. The CASA table export is the extra feature that's been disabled. @JSKenyon if you can take a look what's up there, that'd be great.

@caseyjlaw another plotting option (for parmdb files) are the plot-xxx-solutions scripts in CubiCal.

JSKenyon commented 3 years ago

There are definitely still problems to be solved. I removed all the checks and run into conformance errors. I am doing a little digging now to see if I can figure out if there is an easy fix.

JSKenyon commented 3 years ago

Ah - I can see where at least one conformance error comes from. It looks like some of the subtables in the casa gain table end up with the wrong number of rows e.g. 40 rows get added, but then a putcol is attempted with 28 rows. Trying to figure out why it happens.

JSKenyon commented 3 years ago

@bennahugo I think you may be better suited to chase this further. What I have found is that the blank table actually contains a non-zero number of rows. Thus, when we do table.addrows, we actually end up with excess rows and conformance errors when we attempt to write to the table. I am not sure if the blank table just needs to be replaced/fixed (something you are more familiar with than me).

JSKenyon commented 3 years ago

For reference, the problems seem to be caused by code like: https://github.com/ratt-ru/CubiCal/blob/57740a4c35396d043ecdd38fed2e08601db44788/cubical/database/casa_db_adaptor.py#L61

bennahugo commented 3 years ago

I'll take a look after I released a new ddf and kms. It could be that this version of casacore needs the rows removed first

On Tue, 30 Mar 2021, 10:06 JSKenyon, @.***> wrote:

For reference, the problems seem to be caused by code like:

https://github.com/ratt-ru/CubiCal/blob/57740a4c35396d043ecdd38fed2e08601db44788/cubical/database/casa_db_adaptor.py#L61

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/440#issuecomment-810009789, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VGTYKUTZGIMXVR4WTTGGBATANCNFSM4YS74PZA .

JSKenyon commented 3 years ago

Just a heads up that #448 may fix this for anyone wanting to take it for a spin.