search

ratt-ru / xova

dask-ms/codex-africanus MS averaging tool
Other
0 stars 2 forks source link

Baseline Dependent Time and Channel Averaging. #12

Closed sjperkins closed 2 years ago

sjperkins commented 4 years ago

Needs the following PR's:

to work.

sjperkins commented 4 years ago

DDID and SPW tables still not set up, also need to correctly populate DATA_DESC_ID in the main table.

sjperkins commented 3 years ago

@osmirnov Here's a command line that worked for me locally:

$ xova bda ~/data/1527016443_sdp_l0.full_1284.full_pol.ms --force -o /tmp/avg.ms -dc DATA:DATA MODEL_DATA:MODEL_DATA
o-smirnov commented 3 years ago

Hang on, is it multiple arguments to -dc?

sjperkins commented 3 years ago

Hang on, is it multiple arguments to -dc?

No just one should do it, the log file should tell you about the DATA columns it's going to average and their destination:

(xova) sperkins@simon-Y700:~/work/ska/code/xova$ xova bda ~/data/1527016443_sdp_l0.full_1284.full_pol.ms --force -o /tmp/avg.ms -dc DATA:DATA MODEL_DATA:MODEL_DATA
2020-10-20 13:11:01 | INFO     | app:execute | xova version 0.1.1
2020-10-20 13:11:01 | INFO     | app:execute | xova bda /home/sperkins/data/1527016443_sdp_l0.full_1284.full_pol.ms --force -o /tmp/avg.ms -dc DATA:DATA MODEL_DATA:MODEL_DATA
2020-10-20 13:11:01 | INFO     | arguments:log_args | Configuration:
2020-10-20 13:11:01 | INFO     | arguments:log_args | Baseline-Dependant Time and Channel Averaging
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Averaging '/home/sperkins/data/1527016443_sdp_l0.full_1284.full_pol.ms' to '/tmp/avg.ms'
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Acceptable decorrelation factor 0.99
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Maximum Field of View 15.0 degrees
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Minimum number of channels in output 1. This will be rounded up to the nearest integer factorisation of the number of input channels
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Averaging DATA into DATA
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Averaging MODEL_DATA into MODEL_DATA
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Approximately 10000 rows will be averaged as independent chunks. Unique times will not be split across chunks.
2020-10-20 13:11:01 | INFO     | arguments:log_args |   Grouping 4 averaged row chunks together for writing
2020-10-20 13:11:01 | WARNING  | arguments:log_args |   Discarding Auto-Correlations
2020-10-20 13:11:01 | WARNING  | arguments:log_args |   Overwriting '/tmp/avg.ms'
2020-10-20 13:11:01 | WARNING  | arguments:log_args |   Overriding FLAG_ROW values with any(FLAG, axis=(1,2))
2020-10-20 13:11:02 | INFO     | chunking:dataset_chunks | Dataset Chunking: (r)ow - (t)imes - (s)econds
2020-10-20 13:11:02 | INFO     | chunking:dataset_chunks | Dataset 0: (9450r,15t,119.9s), (9450r,15t,119.9s), (3780r,6t,48.0s)
2020-10-20 13:11:02 | INFO     | chunking:dataset_chunks | Dataset 1: (9450r,15t,119.9s), (9450r,15t,119.9s), (4410r,7t,56.0s)
2020-10-20 13:11:02 | INFO     | chunking:dataset_chunks | Dataset 2: (9450r,15t,119.9s), (5040r,8t,64.0s)
o-smirnov commented 3 years ago

Yep, I have it right now.

I used multiple -dc's the first time, and I think it only effected the last one. Which makes sense.

sjperkins commented 3 years ago

@bennahugo This branch now points at codex-africanus master. I'd recommend basing your UVW fix on this xova branch.

bennahugo commented 3 years ago

Am doing that. Thanks

bennahugo commented 3 years ago

Hmm normal time_and_channel averager is broken in this branch

Traceback (most recent call last):
  File "/home/hugo/workspace/venvxova/bin/xova", line 33, in <module>
    sys.exit(load_entry_point('xova', 'console_scripts', 'xova')())
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 36, in main
    Application(sys.argv[1:]).execute()
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 79, in execute
    viscolumn=args.data_column)
  File "/home/hugo/workspace/xova/xova/apps/xova/averaging.py", line 255, in average_main
    **kwargs)
TypeError: time_and_channel() got an unexpected keyword argument 'vis'
bennahugo commented 3 years ago

nvm found it and fixed it in xova

sjperkins commented 3 years ago

Hmm normal time_and_channel averager is broken in this branch

Traceback (most recent call last):
  File "/home/hugo/workspace/venvxova/bin/xova", line 33, in <module>
    sys.exit(load_entry_point('xova', 'console_scripts', 'xova')())
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 36, in main
    Application(sys.argv[1:]).execute()
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 79, in execute
    viscolumn=args.data_column)
  File "/home/hugo/workspace/xova/xova/apps/xova/averaging.py", line 255, in average_main
    **kwargs)
TypeError: time_and_channel() got an unexpected keyword argument 'vis'

Ah, this is because the inputs got changed from vis to visibilities. I don't think I've done the same with the output. Would it be OK if I put a quick PR into codex and you handled the change on the xova end?

bennahugo commented 3 years ago

Sure

On Fri, 12 Mar 2021, 16:03 Simon Perkins, @.***> wrote:

Hmm normal time_and_channel averager is broken in this branch

Traceback (most recent call last): File "/home/hugo/workspace/venvxova/bin/xova", line 33, in sys.exit(load_entry_point('xova', 'console_scripts', 'xova')()) File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 36, in main Application(sys.argv[1:]).execute() File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 79, in execute viscolumn=args.data_column) File "/home/hugo/workspace/xova/xova/apps/xova/averaging.py", line 255, in average_main **kwargs) TypeError: time_and_channel() got an unexpected keyword argument 'vis'

Ah, this is because the inputs got changed from vis to visibilities. I don't think I've done the same with the output. Would it be OK if I put a quick PR into codex and you handled the change on the xova end?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/xova/pull/12#issuecomment-797509302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6VRH6DL5VYJKIPKHOTTDINKTANCNFSM4OL7B47A .

sjperkins commented 3 years ago

Hmm normal time_and_channel averager is broken in this branch

Traceback (most recent call last):
  File "/home/hugo/workspace/venvxova/bin/xova", line 33, in <module>
    sys.exit(load_entry_point('xova', 'console_scripts', 'xova')())
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 36, in main
    Application(sys.argv[1:]).execute()
  File "/home/hugo/workspace/xova/xova/apps/xova/app.py", line 79, in execute
    viscolumn=args.data_column)
  File "/home/hugo/workspace/xova/xova/apps/xova/averaging.py", line 255, in average_main
    **kwargs)
TypeError: time_and_channel() got an unexpected keyword argument 'vis'

Ah, this is because the inputs got changed from vis to visibilities. I don't think I've done the same with the output. Would it be OK if I put a quick PR into codex and you handled the change on the xova end?

Change is in master

bennahugo commented 3 years ago

This is ready to be tested in force now. I will synthesize some images and compare the astrometry week after this coming one

bennahugo commented 3 years ago

I think this still misses -mt MAX_TIME (per single baseline integration) as we discussed (to control averaging on shortest spacings). There is only one for MIN_NCHAN? @sjperkins

edit: or is this what this is

  -t TIME_BIN_SECS, --time-bin-secs TIME_BIN_SECS
                        Number of seconds to average into a single bin

Perhaps the documentation needs to be clarified?

bennahugo commented 3 years ago

Not having any success with bda @sjperkins. I double checked that this is not due to the changes I introduced. before and after bda averaging (normal averager works): image

I ran (this disables UVW decomputation btw): xova bda msdir/1491291289.1ghz.1.1ghz.4hrs.ms -fov 2.0 -d 0.99 -dc CORRECTED_DATA --force -o msdir/1491291289.1ghz.1.1ghz.4hrs.xova.ms --average-uvw-coordinates

Maybe this is because the dataset is missing rows - been split already?

pip freeze

aiocontextvars==0.2.2
appdirs==1.4.4
astroid==2.5.1
astropy==4.1
attrs==20.3.0
backcall==0.2.0
-e git+git@github.com:ska-sa/codex-africanus.git@36289aa525871c393f11e212e282b7f39bf2b979#egg=codex_africanus
contextvars==2.4
dask==2021.3.0
dask-ms==0.2.6
decorator==4.4.2
future==0.18.2
immutables==0.15
importlib-metadata==3.7.2
iniconfig==1.1.1
ipdb==0.13.6
ipython==7.16.1
ipython-genutils==0.2.0
isort==5.7.0
jedi==0.18.0
lazy-object-proxy==1.5.2
llvmlite==0.35.0
loguru==0.5.3
mccabe==0.6.1
numba==0.52.0
numpy==1.19.5
packaging==20.9
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
pkg-resources==0.0.0
pluggy==0.13.1
prompt-toolkit==3.0.17
ptyprocess==0.7.0
py==1.10.0
Pygments==2.8.1
pylint==2.7.2
pyparsing==2.4.7
pytest==6.2.2
python-casacore==3.4.0
PyYAML==5.4.1
scipy==1.5.4
six==1.15.0
toml==0.10.2
toolz==0.11.1
traitlets==4.3.3
typed-ast==1.4.2
typing-extensions==3.7.4.3
wcwidth==0.2.5
wrapt==1.12.1
-e git+git@github.com:ska-sa/xova.git@a5610f202f89a6f2f01364efa0ccde7aacdbbf5a#egg=xova
zipp==3.4.1

I have pushed my parsets and calibrated test dataset to the ftp: ftp://elwood.ru.ac.za/pub/hugo/1491291289.1ghz.1.1ghz.4hrs.calibrated.tar.gz

sjperkins commented 3 years ago

I think this still misses -mt MAX_TIME (per single baseline integration) as we discussed (to control averaging on shortest spacings). There is only one for MIN_NCHAN? @sjperkins

edit: or is this what this is

  -t TIME_BIN_SECS, --time-bin-secs TIME_BIN_SECS
                        Number of seconds to average into a single bin

Perhaps the documentation needs to be clarified?

Yes that it.

JSKenyon commented 3 years ago

@bennahugo You could try running with a decorrelation factor of 1 and see what happens. Should be an identity operation so if you still get garbage out, then at least we know the problem is not with the averaging itself.

bennahugo commented 3 years ago

I can - it may just be an indexing issue upon missing rows? I presume up until this point people have been going from 1GC calibrated full datasets

sjperkins commented 3 years ago

I'm a bit surprised. Here are the changes from the last known working commit https://github.com/ska-sa/codex-africanus/pull/173/commits/a1323e05861d49d844994207d738419b04d2022c:

Aside from test case changes, the only significant change was a renaming of the BDA mapping function.

Maybe this is because the dataset is missing rows - been split already?

This may be something to investigate. What exactly did the spllt operation do to the data. Are there disjoint time samples (i.e. the TIME's and INTERVAL's don't line up a on grid?

bennahugo commented 3 years ago

some rows are missing so the grid has holes in it yes. Also of course because it is averaged not all centroids are exactly 8s apart - this is a nice little dataset I use to stress test implementations. It has yielded many a bug to me before :). let me test your last known working commit

sjperkins commented 3 years ago

some rows are missing so the grid has holes in it yes. Also of course because it is averaged not all centroids are exactly 8s apart

An irregular grid should be fine. I'd be somewhat concerned about holes in the grid, see my previous comments on the matter here: https://github.com/ska-sa/xova/issues/2#issuecomment-577638414

bennahugo commented 3 years ago

The latter case is because the data was averaged with fully flagged rows removed. Unfortunately most data take this form. If we don't support this case we should raise an exception though - it is seriously crippling for our idea to archive bda data though.

Could XOVA not pad the data back up?

sjperkins commented 3 years ago

The latter case is because the data was averaged with fully flagged rows removed. Unfortunately most data take this form. If we don't support this case we should raise an exception though - it is seriously crippling for our idea to archive bda data though.

Once rows have been removed from the dataset, further averaging will create invalid values for columns with nominal definitions (TIME and INTERVAL especially) because by definition nominal columns contain "missing and flagged data". In my opinion flagged rows should not be removed if you want to average further -- I understand that people do it, but IMO it is a misuse of the tool.

sjperkins commented 3 years ago

Out of interest, do similar issues occur if one averages a CASA MS that has (1) been averaged (2) followed by removal of flagged rows?

bennahugo commented 3 years ago

Sorry I misread. There is no tool to do that so I'm not sure.

sjperkins commented 3 years ago

Sorry I misread. There is no tool to do that so I'm not sure.

Well let me put it another way: What happens if mstransform/split averaging is reapplied on averaged data with flagged rows removed?

I.e. average it to 4s intervals, then re-average it to 16s intervals?

I imagine people just don't do that -- they probably average once.

bennahugo commented 3 years ago

Sorry I misread. There is no tool to do that so I'm not sure.

Well let me put it another way: What happens if mstransform/split averaging is reapplied on averaged data with flagged rows removed?

I.e. average it to 4s intervals, then re-average it to 16s intervals?

I imagine people just don't do that -- one probably averages once.

In principle nothing of note happens. I have averaged down systematically before and things continue to work as expected. The total exposure time is just systematically increased per bin

bennahugo commented 3 years ago

I'm not sure why nominal times cannot be padded back up - these are just missing baselines for a given timeslot. Those are easily added back in. If the entire scan is out then there are no padding. So the total size should be np.unique(TIME) nbl nfield * nddid - exactly like what we've done for Montblanc in DDFacet and Cubical?

sjperkins commented 3 years ago

I'm not sure why nominal times cannot be padded back up - these are just missing baselines for a given timeslot. Those are easily added back in. If the entire scan is out then there are no padding. So the total size should be np.unique(TIME) nbl nfield * nddid - exactly like what we've done for Montblanc in DDFacet and Cubical?

In the general case, the grid is not regular.

sjperkins commented 3 years ago

https://en.wikipedia.org/wiki/Unstructured_grid vs https://en.wikipedia.org/wiki/Regular_grid, for instance.

sjperkins commented 3 years ago

Just turn on keepflags here https://casa.nrao.edu/docs/TaskRef/mstransform-task.html.

bennahugo commented 3 years ago

Most of my archived datasets has already been averaged without keeping rows. Unfortunately I have no galactic plane data that is in the full spectro-temporal resolution. To test BDA I will have to start from scratch. I'm mostly interested in the effect of BDA averaging on diffuse emission off axis, so I will need to calibrate from scratch to test it. I don't have the bandwidth to do this at the moment

sjperkins commented 3 years ago

Most of my archived datasets has already been averaged without keeping rows. Unfortunately I have no galactic plane data that is in the full spectro-temporal resolution. To test BDA I will have to start from scratch. I'm mostly interested in the effect of BDA averaging on diffuse emission off axis, so I will need to calibrate from scratch to test it. I don't have the bandwidth to do this at the moment

Just to summarise my thoughts from today's meeting: If rows are missing in a chunk of data, it is possible to impute/interpolate reasonable values for TIME and INTERVAL (Haven't considered other columns throughout the MS spec) if the missing row lies between other rows.

The challenging case is when rows are missing at either end of a chunk: Two values are required to impute/interpolate, but this is not possible because the end row is missing (not flagged).

Using mstransform with keepflags=True doesn't remove flagged data, so the above case is never reached.