search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

TypeError: unsupported format string passed to numpy.ndarray.__format__ #434

Closed viralp closed 3 years ago

viralp commented 3 years ago

Hi,

I am getting the following error

INFO 14:42:54 - solver [x16] [1.0/216.8 6.6/620.5 167.7Gb] DD kernels are <module 'cubical.kernels.full_complex' from '/home/parekh/CubiCal/cubical/kernels/full_complex.py'> <module 'cubical.kernels.full_complex' from '/home/parekh/CubiCal/cubical/kernels/full_complex.py'> ERROR 14:43:26 - solver [x09] [1.4/222.7 7.0/675.7 167.7Gb] Solver for tile 0 chunk D0T1F2 failed with exception: unsupported format string passed to numpy.ndarray.format INFO 14:43:26 - solver [x09] [1.4/222.7 7.0/675.7 167.7Gb] Traceback (most recent call last): File "/home/parekh/CubiCal/cubical/solver.py", line 926, in run_solver corr_vis = solver_machine.run() File "/home/parekh/CubiCal/cubical/solver.py", line 762, in run SolveOnly.run(self) File "/home/parekh/CubiCal/cubical/solver.py", line 729, in run self.sol_opts, label=self.label) File "/home/parekh/CubiCal/cubical/solver.py", line 104, in _solve_gains gm.precompute_attributes(obser_arr, model_arr, flags_arr, inv_var_chan) File "/home/parekh/CubiCal/cubical/machines/jones_chain_machine.py", line 128, in precompute_attributes term.precompute_attributes(data_arr, model_arr, flags_arr, inv_var_chan) File "/home/parekh/CubiCal/cubical/machines/complex_2x2_machine.py", line 190, in precompute_attributes super(Complex2x2Gains, self).precompute_attributes(data_arr, model_arr, flags_arr, noise) File "/home/parekh/CubiCal/cubical/machines/interval_gain_machine.py", line 504, in precompute_attributes self._report_gain_flags(low_snr, "on low SNR", "your max-prior-error settings", self.low_snr_warn) File "/home/parekh/CubiCal/cubical/machines/interval_gain_machine.py", line 563, in _report_gain_flags str(d), percflagged[d]) for d in bad_dirs]), File "/home/parekh/CubiCal/cubical/machines/interval_gain_machine.py", line 563, in str(d), percflagged[d]) for d in bad_dirs]), TypeError: unsupported format string passed to numpy.ndarray.format ERROR 14:43:37 - solver [x08] [1.4/224.5 7.0/693.6 167.7Gb] Solver for tile 0 chunk D0T1F1 failed with exception: unsupported format string passed to numpy.ndarray.format ddcal_0.log

o-smirnov commented 3 years ago

This is like the zombie bug from hell. I'm sure we've fixed it at least twice (remember, @bennahugo?). And it was only ever a debugging message anyway!

bennahugo commented 3 years ago

I recall. But it can be worked around by disabling verbose logging if I remember correctly.

On Thu, 04 Feb 2021, 11:01 Oleg Smirnov, notifications@github.com wrote:

This is like the zombie bug from hell. I'm sure we've fixed it at least twice (remember, @bennahugo https://github.com/bennahugo?). And it was only ever a debugging message anyway!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/434#issuecomment-773146177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6SRPM5LYA7WFTSTJE3S5JO55ANCNFSM4XCKJRPA .

viralp commented 3 years ago

@Ben you meant to disable --log-verbose solver with 0?

bennahugo commented 3 years ago

yes

On Thu, Feb 4, 2021 at 11:53 AM Viral Parekh notifications@github.com wrote:

@ben https://github.com/ben you meant to disable --log-verbose solver with 0?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/434#issuecomment-773179970, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6THESIN3KTSMICIX2LS5JVAVANCNFSM4XCKJRPA .

--

Benjamin Hugo

PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University

Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

bennahugo commented 3 years ago

This zombie bug from hell is STILL with us in v1.5.11... FFS. I will look into it tomorrow as this has now reached critical itch level

JSKenyon commented 3 years ago

I am happy to help but I cannot do much without a way to reproduce.

o-smirnov commented 3 years ago

@bennahugo can you please replace percflagged[d] with float(percflagged[d]) in the offending line and try again? I think we keep complaining about it but never bother to fix it.

bennahugo commented 3 years ago

Thanks yes I'm busy looking at a reproducer for this locally. I think it is as easy as drawing a region where nothing was cleaned, ie no model exist. I think we are dealing with a scalar array and then that message tries to format it as float. To be confirmed though.

On Wed, 18 Aug 2021, 11:39 Oleg Smirnov, @.***> wrote:

@bennahugo https://github.com/bennahugo can you please replace percflagged[d] with float(percflagged[d]) in the offending line and try again? I think we keep complaining about it but never bother to fix it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/434#issuecomment-900970662, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WEDGEHX66FZEC6P4LT5N5UFANCNFSM4XCKJRPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

bennahugo commented 3 years ago

Alright I have a reproducer - empty model is not the answer - it skipps and runs through in this case... severe clips on the prior posterior SNR is the answer along with solver==2 verbosity.