Plots are broken

[JSKenyon]

I keep having problems whenever plots are enabled:

Exiting with exception: ComplexWarning(Casting complex values to real discards the imaginary part) Traceback (most recent call last): File "/home/jonathan/CubiCal/cubical/main.py", line 399, in main cubical.plots.ifrgains.make_ifrgain_plots(solver.ifrgain_machine.reload(), ms, GD, basename) File "/home/jonathan/CubiCal/cubical/plots/ifrgains.py", line 199, in make_ifrgain_plots norm_igs = [ (pq, _normifrgain(rr),_normifrgain(ll)) for pq, rr, ll in valid_igs ] File "/home/jonathan/CubiCal/cubical/plots/ifrgains.py", line 21, in _normifrgain offset = abs(rr[rr != 1]) File "/home/jonathan/venv/setupenv/local/lib/python2.7/site-packages/numpy/ma/core.py", line 3016, in __array_finalize__ self._fill_value = _check_fill_value(self._fill_value, self.dtype) File "/home/jonathan/venv/setupenv/local/lib/python2.7/site-packages/numpy/ma/core.py", line 476, in _check_fill_value fill_value = np.array(fill_value, copy=False, dtype=ndtype) ComplexWarning: Casting complex values to real discards the imaginary part

For now I am going to turn plots off in the default parset so that I can do a release. Note that setting the bbc-plot option to False doesn't suppress the error. I had to set out-plots to False to get around this.

[o-smirnov]

Ah ok I started reviewing https://github.com/ratt-ru/CubiCal/pull/235 before I saw this, hence my question there.

As a matter of philosophy, I'd prefer plots on by default. Otherwise people never look at them, while they may contain useful insights. So let me take a look at this error, maybe we can fix it for this release.

I suspect it may be a numpy bug in your particular version. Why is offset = abs(rr[rr != 1]) (for complex rr) problematic? And why is it exiting with the exception, when normal behaviour is to print the ComplexWarning and move on?

Also, just for the sake of robustness, I'll put try-except clauses around plotting code, so that future numpy or matplotlib hissy fits don't cause the whole process to fall down.

[o-smirnov]

Could you try this snippet in ipython on your installation, to see if you get a similar error:

import numpy as np
a=np.array([1+0j,2+0j])
abs(a[a==1])

[JSKenyon]

That seems to work without issue.

[o-smirnov]

But I assume you get that error with the built-in d147 test?

[JSKenyon]

No - my own tiny WSRT test. I guess it could be a broken MS, but I am pretty sure I have seen the problem before.

[o-smirnov]

Can you send me the MS and parset?

[JSKenyon]

Not managing to reproduce it elsewhere, so don't worry. Probably a broken MS. Will adjust the PR.

[o-smirnov]

I'd still like to see the MS. CubiCal should be more robust about these things. And I'd like to put a guard around the plotting code.

[JSKenyon]

On Crosby in directory named broken_ms.

[o-smirnov]

Seems to run fine for me:

 - 17:02:02 - plots              [0.2/0.3 1.2/1.3 0.8Gb] generating plots for suggested baseline-based corrections (BBCs)
 - 17:02:05 - plots              [0.2/0.3 1.2/1.3 0.8Gb] saved plot cubical.bbc-diag.png
 - 17:02:09 - plots              [0.3/0.4 1.3/1.3 0.8Gb] saved plot cubical.bbc-diag-ant.png
 - 17:02:09 - main               [0.3/0.4 1.3/1.3 0.8Gb] completed successfully

So I'm still inclined to blame your numpy version. But in any case I'll add a guard around the plotting code.

[o-smirnov]

And why is it exiting with the exception, when normal behaviour is to print the ComplexWarning and move on?

Haha, this is why: https://github.com/ratt-ru/CubiCal/blob/master/cubical/main.py#L397

I've deliberately upgraded them into an error! Looks like I was chasing down these ComplexWarnings at some point. Obviously, not yet entirely successfully.

[o-smirnov]

@JSKenyon could you pull the fix-parset branch and give it a try please? I've added an option called --debug-escalate-warnings. If False (default), your ComplexWarning should be reported but not cause the whole house to fall down. If True, it should be back to the current behaviour of falling over.

If this works, I should add some logic to madmax to do the same -- it's caused us many an error from matplotlib, usually when trying to plot degenerate data. I'd like it to just report a plot failure and move on.

[JSKenyon]

The error seems to have disappeared. Should be sorted for now.

[o-smirnov]

Ha, now I'm seeing it myself...

[o-smirnov]

I still reckon it's a bug in numpy, specifically in how masked arrays implement the abs() built-in. Replacing abs(rr[rr != 1]) with np.abs(rr[rr != 1]) makes it go away.