search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

The selected DDID/frequencies in this MS are not laid out monotonically #464

Closed AKHughes1994 closed 1 year ago

AKHughes1994 commented 2 years ago

I'm getting this error when running CubiCal on some VLA data, and it's killing all runs almost immediately,

image

I'm not sure if I'm misinterpreting the error (I'm not exactly sure what DDID stands for) but my frequencies are absolutely laid out Monotonically, it's just 8-bit S-Band VLA data (I flagged the first 3 Spws due to RFI) but the data increases from CtrFreq = 2435MHz to 3947MHz from spw 0 to 12.

Any advice on how to Debug this issue? Cheers, Andrew

AKHughes1994 commented 2 years ago

I will also note that in my parset file i have, --sel-ddid=None

IanHeywood commented 2 years ago

DDID = Data Description ID, which is most cases is just synonymous with spectral window.

Do the SPWs all have equivalent channel widths? What does the output of CASA's listobs say?

You could also try iterating over SPWs sequentially with CubiCal, using --sel-ddid.

AKHughes1994 commented 2 years ago

The output is pretty typical of VLA observations,

image

Excluding flagging, each SPW has 64x 2 MHz = 128MHz bandwidth, the lower the SPW ID the lower the central frequency. I was playing around with --sel-ddid and it works with 0\~4 and 5\~12. Since I flagged the first three SPWS this actually corresponds to the two base-bands of the original 16 SPW VLA observations, despite the transition in frequency between base-bands being monotonic.

AKHughes1994 commented 2 years ago

Is there a way to append the two solution intervals for a single directory? If I run 0~4, then 5~12 refuses to run since my directory already exists, or do I just rename the directory?

JSKenyon commented 2 years ago

Hi @AKHughes1994! I am not entirely sure what would be going wrong here. The error message you are seeing comes from here:

https://github.com/ratt-ru/CubiCal/blob/59f28596150d15e4d673fd07999050e770d00fb2/cubical/data_handler/ms_data_handler.py#L499-L504

It is possible that this check is slightly wrong, but it will be tricky to fix without access to your data.

Is there a way to append the two solution intervals for a single directory? If I run 0~4, then 5~12 refuses to run since my directory already exists, or do I just rename the directory?

You could just output the second set to a second directory. I appreciate this may not be convenient though (depending on what further processing you want to do).

AKHughes1994 commented 2 years ago

If you are interested in the data

You should be able to download the ms file from this link,

https://drive.google.com/file/d/1zyXpEbECz4J_vXIC1AljdXcHFItWEJNU/view?usp=sharing

On Tue, Jun 14, 2022 at 12:57 AM JSKenyon @.***> wrote:

Hi @AKHughes1994 https://github.com/AKHughes1994! I am not entirely sure what would be going wrong here. The error message you are seeing comes from here:

https://github.com/ratt-ru/CubiCal/blob/59f28596150d15e4d673fd07999050e770d00fb2/cubical/data_handler/ms_data_handler.py#L499-L504

It is possible that this check is slightly wrong, but it will be tricky to fix without access to your data.

Is there a way to append the two solution intervals for a single directory? If I run 04, then 512 refuses to run since my directory already exists, or do I just rename the directory?

You could just output the second set to second directory. I appreciate this may not be convenient though (depending on what further processing you want to do).

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/464#issuecomment-1154791347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL3FQF63AS5QOMSL3RFCCR3VPAUOLANCNFSM5YVHLFOA . You are receiving this because you were mentioned.Message ID: @.***>

JSKenyon commented 2 years ago

Great! Could you please provide your parset? That should be enough for me to figure it out.

On Tue, 14 Jun 2022, 20:30 AKHughes1994, @.***> wrote:

If you are interested in the data

You should be able to download the ms file from this link,

https://drive.google.com/file/d/1zyXpEbECz4J_vXIC1AljdXcHFItWEJNU/view?usp=sharing

On Tue, Jun 14, 2022 at 12:57 AM JSKenyon @.***> wrote:

Hi @AKHughes1994 https://github.com/AKHughes1994! I am not entirely sure what would be going wrong here. The error message you are seeing comes from here:

https://github.com/ratt-ru/CubiCal/blob/59f28596150d15e4d673fd07999050e770d00fb2/cubical/data_handler/ms_data_handler.py#L499-L504

It is possible that this check is slightly wrong, but it will be tricky to fix without access to your data.

Is there a way to append the two solution intervals for a single directory? If I run 04, then 512 refuses to run since my directory already exists, or do I just rename the directory?

You could just output the second set to second directory. I appreciate this may not be convenient though (depending on what further processing you want to do).

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/464#issuecomment-1154791347, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AL3FQF63AS5QOMSL3RFCCR3VPAUOLANCNFSM5YVHLFOA

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/464#issuecomment-1155581421, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSHDWMHB4DXR7HQYR6V663VPDFTRANCNFSM5YVHLFOA . You are receiving this because you commented.Message ID: @.***>

JSKenyon commented 2 years ago

Ok, I have managed to reproduce. The issue is that SPW4 and SPW5 overlap. Largest CHAN_FREQ in SPW4 is 3.010GHz, smallest in SPW5 is 2.988GHz. CubiCal concatenates along the frequency axis and doesn't know how to handle this situation. Depending on what you are doing, you could try QuartiCal. It processes each SPW separately (which is sensible for the VLA).

Alternatively, you can process the your data in two passes. You could use --sel-ddid=0~4 and --sel-ddid=5~12 to skirt around the problem.

Another approach would be to split the problem into two measurement sets and simply image them simultaneously.

Hope this helps - it is not so much a bug as a limitation in CubiCal.

o-smirnov commented 2 years ago

Anyone have suggestions for a better error message in this case?

JSKenyon commented 2 years ago

"The selected spectral windows (DDIDs) in this MS are not laid out monotonically and/or overlap. This is not currently supported. Consider using --sel-ddid to process a single spectral window/range of spectral windows independently."

AKHughes1994 commented 2 years ago

Thank you for the clarification, just breaking it into two interactions is a very easy work around!

On Fri, Jun 17, 2022 at 1:00 AM JSKenyon @.***> wrote:

"The selected spectral windows (DDIDs) in this MS are not laid out monotonically and/or overlap. This is not currently supported. Consider using --sel-ddid to process a single spectral window/range of spectral windows independently."

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/464#issuecomment-1158558906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL3FQF6I53J432K7SRICEZTVPQPCTANCNFSM5YVHLFOA . You are receiving this because you were mentioned.Message ID: @.***>

JSKenyon commented 1 year ago

Closing as I think this is "resolved".