search

ratt-ru / shadeMS

Rapid Measurement Set plotting with dask-ms and datashader
21 stars 6 forks source link

Crash when plotting AMP vs. FREQ over 8 SPWs #94

Open Jordatious opened 3 years ago

Jordatious commented 3 years ago

I'm getting the error below when calling shadeMS in the following way:

shadems -j 32 --xaxis FREQ --yaxis MODEL_DATA:amp 1587795059_sdp_l0.J03323-28075.ms

I managed to get it working when plotting amplitude as a function of time, but not as a function of frequency, as with this call.

Here's some info about the data in case it's useful. There are discontinuities between SPWs, either side of RFI bands. And then obviously the number of channels isn't consistent between SPWs. I'm not sure if that's problematic or not.

190G    1587795059_sdp_l0.J03323-28075.ms
nant: 59
nbaseline: 1711
Columns present: ['UVW', 'FLAG', 'FLAG_CATEGORY', 'WEIGHT', 'SIGMA', 'ANTENNA1', 'ANTENNA2', 'ARRAY_ID', 'DATA_DESC_ID', 'EXPOSURE', 'FEED1', 'FEED2', 'FIELD_ID', 'FLAG_ROW', 'INTERVAL', 'OBSERVATION_ID', 'PROCESSOR_ID', 'SCAN_NUMBER', 'STATE_ID', 'TIME', 'TIME_CENTROID', 'DATA', 'WEIGHT_SPECTRUM', 'SIGMA_SPECTRUM', 'MODEL_DATA', 'CORRECTED_DATA', 'DIR1_DATA']
nspw: 8
Freqs: 880-1419 MHz
Channel width: 836 kHz
Freqs (spw 0): 880.4119873046875-932.2401123046875 MHz (nchan: 63)
Freqs (spw 1): 960.4268798828125-1008.9112548828125 MHz (nchan: 59)
Freqs (spw 2): 1010.4263916015625-1058.9107666015625 MHz (nchan: 59)
Freqs (spw 3): 1060.4259033203125-1108.9102783203125 MHz (nchan: 59)
Freqs (spw 4): 1110.4254150390625-1162.2535400390625 MHz (nchan: 63)
Freqs (spw 5): 1301.4110107421875-1331.5047607421875 MHz (nchan: 37)
Freqs (spw 6): 1332.4190673828125-1379.2315673828125 MHz (nchan: 57)
Freqs (spw 7): 1380.4071044921875-1418.8602294921875 MHz (nchan: 47)
nchans: 444
fields: ['J03323-28075']
npol: 2
Tobs: 9.940239161427598 hours
Dump time: 7.9966169712
nscans: 22

Log:

2021-03-10 11:58:58 - shadems - INFO - using colourmap colorcet.bkr
2021-03-10 11:58:59 - shadems - INFO - using colourmap cmasher.pride
2021-03-10 11:58:59 - shadems - INFO - using colourmap colorcet.glasbey_dark
2021-03-10 11:58:59 - shadems - INFO - /idia/users/jcollier/software/shadems-venv/bin/shadems -j 32 --xaxis FREQ --yaxis MODEL_DATA:amp 1587795059_sdp_l0.J03323-28075.ms
2021-03-10 11:58:59 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:58:59 - shadems - INFO - : MS 1587795059_sdp_l0.J03323-28075.ms contains 44937704 rows
2021-03-10 11:58:59 - shadems - INFO - :   (8, 63) spectral windows and channels
2021-03-10 11:58:59 - shadems - INFO - :   1 fields: J03323-28075
2021-03-10 11:59:05 - shadems - INFO - :   22 scans: 3 5 7 9 11 13 15 17 19 21 25 27 29 31 33 35 37 39 41 43 47 49
2021-03-10 11:59:17 - shadems - INFO - :   59/59 antennas: 0:m000 1:m001 2:m002 3:m003 4:m004 5:m005 6:m006 7:m007 8:m008 9:m009 10:m010 11:m011 12:m012 13:m013 14:m014 15:m015 16:m016 17:m017 18:m018 19:m019 20:m020 21:m021 22:m022 23:m024 24:m026 25:m027 26:m028 27:m029 28:m030 29:m031 30:m032 31:m033 32:m034 33:m035 34:m038 35:m039 36:m040 37:m041 38:m042 39:m044 40:m045 41:m046 42:m047 43:m048 44:m049 45:m050 46:m051 47:m052 48:m053 49:m054 50:m055 51:m056 52:m057 53:m058 54:m059 55:m060 56:m061 57:m062 58:m063
2021-03-10 11:59:23 - shadems - INFO - :   1711/1770 baselines present
2021-03-10 11:59:24 - shadems - INFO - :   corrs/Stokes XX YY I Q
2021-03-10 11:59:24 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:59:24 - shadems - INFO - : Data selected for plotting:
2021-03-10 11:59:24 - shadems - INFO - Antenna(s)       : all
2021-03-10 11:59:24 - shadems - INFO - Baseline(s)      : all except autocorrelations
2021-03-10 11:59:24 - shadems - INFO - Field(s)         : all
2021-03-10 11:59:24 - shadems - INFO - SPW(s)           : all
2021-03-10 11:59:24 - shadems - INFO - Scan(s)          : all
2021-03-10 11:59:24 - shadems - INFO - Channels         : all
2021-03-10 11:59:24 - shadems - INFO - Corr/Stokes      : XX YY
2021-03-10 11:59:24 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:59:24 - shadems - INFO - loading minmax cache from 1587795059_sdp_l0.J03323-28075-minmax-cache.json
2021-03-10 11:59:24 - shadems - INFO - axis: FREQ, range (None, None), discretization None
2021-03-10 11:59:24 - shadems - INFO - axis: amp(MODEL_DATA), corr None, range (None, None), discretization None
2021-03-10 11:59:24 - shadems - INFO -                  : you have asked for 1 plots employing 2 unique datums
2021-03-10 11:59:43 - shadems - INFO - : Indexing MS and building dataframes (44937704 rows, chunk size is 5000)
Traceback (most recent call last):
  File "/idia/users/jcollier/software/shadems-venv/bin/shadems", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/idia/users/jcollier/software/shadeMS/bin/shadems", line 8, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/idia/users/jcollier/software/shadeMS/shade_ms/main.py", line 426, in main
    row_chunk_size=options.row_chunk_size)
  File "/idia/users/jcollier/software/shadeMS/shade_ms/data_plots.py", line 191, in get_plot_data
    value = axis.get_value(group, corr, extras, flag=flag, flag_row=flag_row, chanslice=chanslice)
  File "/idia/users/jcollier/software/shadeMS/shade_ms/data_mappers.py", line 337, in get_value
    self.minmax = min(self.minmax[0], min1) if self.minmax[0] is not None else min1, \
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/array/core.py", line 1622, in __bool__
    return bool(self.compute())
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/base.py", line 283, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/base.py", line 565, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/threaded.py", line 84, in get
    **kwargs
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 487, in get_async
    raise_exception(exc, tb)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 317, in reraise
    raise exc
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/optimization.py", line 963, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/daskms/reads.py", line 176, in getter_wrapper
    return future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/daskms/reads.py", line 38, in ndarray_getcol
    getcolnp(column, result[rr:rr + rl], startrow=rs, nrow=rl)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/casacore/tables/table.py", line 1052, in getcolnp
    return self._getcolvh(columnname, startrow, nrow, rowincr, nparray)
RuntimeError: Table DataManager error: Internal error: StManIndArray::get/put shapes not conforming
srun: error: compute-056: task 0: Exited with exit code 1
srun: Terminating job step 11005.0
Jordatious commented 3 years ago

Btw, when installing shadeMS via pip (following https://github.com/ratt-ru/shadeMS/pull/95), I get the same error.

Jordatious commented 3 years ago

Ok perhaps I should have checked https://github.com/ratt-ru/shadeMS/issues/85... It's a different error that is output though.

o-smirnov commented 3 years ago

Same underlying cause as #85 I'll bet. Good excuse to bite the bullet and sort out SPW support properly. Could you please split out one scan of your MS for me @Jordatious, for testing?

Jordatious commented 3 years ago

Hey @o-smirnov,

Sure! I put the first scan of the target (scan 3) here:

/scratch3/users/oms/1587795059_sdp_l0.J03323-28075-scan3.ms/

Interestingly, I get the same error when I try to plot this manually... I have no idea how plotms managed to get it working...

In [4]: tb.open('1587795059_sdp_l0.J03323-28075.ms')
Out[4]: True

In [5]: dat=tb.query('ANTENNA1==1 AND ANTENNA2==2',columns='MODEL_DATA,TIME')

In [6]: dat.getcol('MODEL_DATA')[0]
2021-03-11 14:25:35 SEVERE  getcol::MODEL_DATA  Exception Reported: Table DataManager error: getArrayColumnCells shape mismatch in column MODEL_DATA
o-smirnov commented 3 years ago

That's not too surprising. getcol() wants to read the column as a (row,chan,corr) array, and since your selection has a "ragged" channel axis, it simply can't.

I suppose plotms iterates over spectral windows to avoid this, so I should just do the same in shadeMS...

Jordatious commented 3 years ago

Ahhh ok that makes sense. You may already know, but this works for me, which you can see corresponds to the channel count from here:

In [9]: dat = tb.query("ANTENNA1==0 AND ANTENNA2==1 AND DATA_DESC_ID==0", columns='MODEL_DATA,TIME')
In [10]: data=dat.getcol('MODEL_DATA')
In [11]: data.shape
Out[11]: (2, 63, 3283)
In [12]: dat2 = tb.query("ANTENNA1==0 AND ANTENNA2==1 AND DATA_DESC_ID==1", columns='MODEL_DATA,TIME')
In [13]: data2=dat2.getcol('MODEL_DATA')
data
In [14]: data2.shape
Out[14]: (2, 59, 3283)
Jordatious commented 3 years ago

Yep so this is working for me when I iterate over the above, and it's quite fast actually. The plotting bit is what takes so long...

Just a quick correction to the above, which I've edited - this was the run where I have all 8 SPWs, without overlap, only avoiding RFI bands, so there's no missing bands. It's the one for which I'm running CubiCal, which was partially successful, and which I'll report back about later. Part of determining why it wasn't 100% successful is plotting the MODEL_DATA and DIR1_DATA columns.