Optimised wsclean predict

[sjperkins]

• [x] Tests added / passed

  $ py.test --flake8 -v -s .

  If the flake8 tests fail, the quickest way to correct this is to run autopep8 and then flake8 to fix the remaining issues.

  $ pip install -U autopep8 flake8
  $ autopep8 -r -i .
  $ flake8 .

[paoloserra]

Below I show 4 regions of a continuum image made from a test .MS with WSclean (left panels) and the line spectrum extracted at the position of the green open circle (right panels). The red line shows the spectrum before continuum subtraction. The white line shows the spectrum after continuum subtraction. The continuum is subtracted by subtracting the MODEL_DATA column written by crystalball from the CORRECTED_DATA column of the .MS.

The plots show that the new, optimised code is doing the right thing: the correct continuum flux density is subtracted at the correct RA,Dec.

We can therefore proceed with measuring the memory requirements and speed of the new code compared to crystalball 0.2.4.

[sjperkins]

Note that I've reduced the default memory requirements from 0.1 to 0.5 as the new code seems to be able to exercise cores with smaller chunks.

Additionally, the source chunk size could probably be increased further with minimal impact at memory

On Wed, 1 Apr 2020, 23:58 paoloserra, notifications@github.com wrote:

Below I show 4 regions of a continuum image made from a test .MS with WSclean (left panels) and the line spectrum extracted at the position of the green open circle (right panels). The red line shows the spectrum before continuum subtraction. The white line shows the spectrum after continuum subtraction. The continuum is subtracted by subtracting the MODEL_DATA column written by crystalball from the CORRECTED_DATA column of the .MS.

The plots show that the new, optimised code is doing the right thing: the correct continuum flux density is subtracted at the correct RA,Dec.

We can therefore proceed with measuring the memory requirements and speed of the new code compared to crystalball 0.2.4.

https://user-images.githubusercontent.com/6591265/78188404-74070e00-7470-11ea-8677-ee6186d3cbb1.png https://user-images.githubusercontent.com/6591265/78188396-70738700-7470-11ea-91e9-8ff9b047e4ab.png

https://user-images.githubusercontent.com/6591265/78190127-b251fc80-7473-11ea-8047-0ba9f81bb35b.png https://user-images.githubusercontent.com/6591265/78190126-b251fc80-7473-11ea-9a4e-72f73acb09bd.png

https://user-images.githubusercontent.com/6591265/78190124-b1b96600-7473-11ea-8e11-a13d06a641e4.png https://user-images.githubusercontent.com/6591265/78190123-b120cf80-7473-11ea-8e55-a01f22833203.png

https://user-images.githubusercontent.com/6591265/78190121-afefa280-7473-11ea-8af1-f94d988af275.png https://user-images.githubusercontent.com/6591265/78190114-ac5c1b80-7473-11ea-9ea0-f367a806dbe4.png

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/paoloserra/crystalball/pull/35#issuecomment-607510373, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA253ZFKQIMPWNY6IYZYKG3RKO2HHANCNFSM4LYWS3PQ .

[paoloserra]

Very happy with this version. I'm OK with releasing the new version of codex-africanus, and then a crystalball 0.2.5.

2020-04-02 19:43:32 | INFO     | budget:get_budget | -------------------------------------------
2020-04-02 19:43:32 | INFO     | budget:get_budget | system RAM = 251.78 GB
2020-04-02 19:43:32 | INFO     | budget:get_budget | nr of logical CPUs = 42
2020-04-02 19:43:32 | INFO     | budget:get_budget | nr sources = 68796
2020-04-02 19:43:32 | INFO     | budget:get_budget | nr rows    = 5744370
2020-04-02 19:43:32 | INFO     | budget:get_budget | nr chans   = 766
2020-04-02 19:43:32 | INFO     | budget:get_budget | nr corrs   = 2
2020-04-02 19:43:32 | INFO     | budget:get_budget | sources per chunk = 840 (auto settings)
2020-04-02 19:43:32 | INFO     | budget:get_budget | rows per chunk    = 84030 (auto settings)
2020-04-02 19:43:32 | INFO     | budget:get_budget | expected memory usage = 50.35 GB

[paoloserra]

Thank you @sjperkins , great improvements!!!