search

ratt-ru / shadeMS

Rapid Measurement Set plotting with dask-ms and datashader
21 stars 6 forks source link

odd performance deterioration with EVN MSs #81

Open o-smirnov opened 3 years ago

o-smirnov commented 3 years ago

@sjperkins, I suspect something about the row ordering is not playing right with dask-ms here. A paltry 5.71e+08 data points (5GB MS) takes ~180 seconds to plot. MeerKAT MSs take the same time at x100 the size, so something is off... image

Simple UV plot, no data column involved even:

shadems n20m3_2Gbps.ms/ -x U -y V -c BASELINE --profile

MS is under /net/simon/home/oms/projects/sms-testing.

sjperkins commented 3 years ago

Could you try increasing the row chunks up an order of magnitude or two?

If the number of channels is low, the amount of data per chunk, and hence aggregation step, will be small.

On Wed, 11 Nov 2020, 17:13 Oleg Smirnov, notifications@github.com wrote:

Assigned #81 https://github.com/ratt-ru/shadeMS/issues/81 to @sjperkins https://github.com/sjperkins.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/81#event-3984256676, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA253ZFF37VRT6QNZLNSMY3SPKSVNANCNFSM4TSC7SLQ .

o-smirnov commented 3 years ago

Already tried that (5000, 50000, 500000), but didn't have any appreciable impact. Note that only the UVW column is read, and it doesn't have channels (and can be read in it entirety in <1s using casacore.tables...)

sjperkins commented 3 years ago

I'll put it on the queue

On Wed, Nov 11, 2020 at 6:53 PM Oleg Smirnov notifications@github.com wrote:

Already tried that (5000, 50000, 500000), but didn't have any appreciable impact. Note that only the UVW column is read, and it doesn't have channels (and can be read in it entirety in <1s using casacore.tables...)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/shadeMS/issues/81#issuecomment-725536137, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA253ZGOBYMOXCOC7ACGUXLSPK6SBANCNFSM4TSC7SLQ .

haavee commented 3 years ago

For reference, I'll link to a casacore issue I created (long) time ago. The results of your experiments seem to indicate this issue may be related: https://github.com/casacore/casacore/issues/800