Closed Kincaidr closed 2 years ago
This error is related to how QurtiCal currently handles the BDA case. It attempts to implicitly restore the time axis to some intermediary regular grid. The reason that this is currently failing is probably the occurrence of several slightly different time values e.g. [71.96951389 71.96955395 71.9695549]
.
The easiest way for me to confirm this if for you to point me to a copy of the data somewhere at Rhodes. Then I can reproduce and debug. It may be that I just need to slightly lower the strictness of the GCD (greatest common divisor) code. There is another possible solution but it is more complicated and I only want to try it as a last resort.
The data is on vault-ike over here:
/vault-ike/kincaid/xova_analysis/xova_real/xova_bda
, MS is called Abell3376_bda_0.7.ms
.
The original pre-bda MS is in the folder before at: /vault-ike/kincaid/xova_analysis/xova_real
and is called Abell3376_raw-Abell3376East-corr.ms
@bennahugo @sjperkins It is worth noting that this touches on a point which I mentioned back in the depths of time. Currently, QuartiCal is trying to figure out the original/intermediate interval values (i.e. the interval which perfectly divides into the all the data in a given chunk). However, this is error-prone as even small variations in the values in the interval column (see above), can make determining this value very difficult. For now I am going to introduce a bit of a hack and round the values. The logic is that if data which contains only one sampling interval (another BDA assumption for now), any BDA row will have some integer multiple of that native interval value. This is more complicated if you have a single scan which contains data dumped with different intervals, but let us cross that bridge when we come to it. The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.
@Kincaidr I am going to test out this hack and see if I run into other problems. Will let you know when I have fixes in place.
The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.
I'm not sure we can assume the original data had a uniform INTERVAL. How about a subtable (BDA_ORIGINAL_GRID) with the original indexing columns?
The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.
I'm not sure we can assume the original data had a uniform INTERVAL. How about a subtable (BDA_ORIGINAL_GRID) with the original indexing columns?
We definitely cannot assume that it had a uniform interval in general, but I do and will continue making that assumption in QuartiCal for now. The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid. That said, I think that if the data has a change in nominal interval e.g. intervals were 8s during the first 30 minutes of a scan and 5s thereafter, it is probably not a good idea to average over the change anyway. A more sensible approach might be to split the scan i.e. we assume some approximately constant fundamental/underlying interval within a scan partition.
I wouldn't object to a subtable which preserves the original indexing columns though. May be generally useful.
The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.
Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?
The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.
Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?
I'm guessing you've looked at a lot of this already though and I am merely interested in whether you found something useful?
The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.
Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?
I'm guessing you've looked at a lot of this already though and I am merely interested in whether you found something useful?
I haven't looked into it extensively. My short term goal was always proof-of-concept - currently I just retile the averaged visibilities onto a regular grid by using the native interval assumption. This is done implicitly along the row axis i.e. has no memory footprint. Channel is simply retiled to the highest frequency resolution in the data. This is done explicitly as two ragged axes becomes problematic without moving into a flattened data regime which I didn't want to consider for PoC code.
While it may be possible to interpolate the BDA data onto an arbitrary grid, it may be a little fraught i.e. what is a sensible interpolation method? And if any bad data ended up in the averaged data, it is possible that interpolating will spread it into more samples. I think that when we last chatted about this I tried to defer the complexity into dask-ms, i.e. implementing an xds_from_bda_storage
type function that could take a parameter like regular=True
to return seemingly regular datasets. This is a big ask though - I was simply unconvinced that the behaviour belongs in QuartiCal.
@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.
This is a big ask though - I was simply unconvinced that the behaviour belongs in QuartiCal.
I'd argue that it might live in codex as the forward and reverse transform will be maintained in the same place.
@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.
Yes, time-only averaging is only supported by xova currently.
@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.
Yes, time-only averaging is only supported by xova currently.
I think it can do channel averaging, but this just hasn't been verified for correctness yet.
@Kincaidr I have solved your first problem and QuartiCal runs through but produces horrible results. I am still debugging.
Closing this for now - should be "fixed" on both the main
and stimelation
branches.
Hi,
I am trying the bda option on Quartical:
input_ms.is_bda=True
for my bda MS with this command:I am getting this error: