Aoflagger performance checks

[jradcliffe5]

Just a quick enquiry.

Javier, have you checked the performance of aoflagger with regards to splitting the flagging into subbands? Especially when you have enough ram to throw all of the dataset in

[jmoldon]

Yes, I made differents tests. These are the conclussions:

• There is no significant difference between processing bands individually and all together if you have enough memory.
• When there is not enough memory then the difference is huge and dividing per bands is much faster if each individual selection can still be in memory (I have seen a factor of x5-7 depending on the size).
• In any of the cases, the memory never gets used so much, so aoflagger seems to be very efficient managing memory, but very bad estimating how much will it need (I suspect it considers all scans when checking if it will have enough memory, but then processes scan by scan, all this in v2.9).

So the default in the pipeline is to divide in bands. The only case I saw it being slower is with very small data sets (about or less than 50GB) in which the overheads for opening the file so many times, num_sources x bands are greater than openning it only num_sources times. For those cases you can select in the inputs file what to do: flag_aoflagger = 1 means process bands individually flag_aoflagger = 2 means process all bands together see documentation

This is an example of the system usage in the case of a big file (225GB fits) and enough memory (256GB). There are 5 sources, three small at the beginning (about first 10 min), one medium phasecal (about 20 min) and the target source (the rest):

Iterations, one spw at a time (1h06m46s)

No iterations, all spw together (1h4m10s)

[jradcliffe5]

Ok great. Thanks Javier! I'll close this issue then. Just to let you know, I'm going to edit the front page for information on easy installing of aoflagger/wsclean using the conda install's I have.